Image processing device, image processing method, and program

ABSTRACT

An image processing device performs a combining process of disposing a second image in a first image captured by a first imaging device, the second image being an image of the imaging field of view of a second imaging device capable of imaging a partial field of view in the imaging field of view of the first imaging device.

TECHNICAL FIELD

The present technology relates to an image processing device, an imageprocessing method, and a program, and relates to a technical field ofimage display processes to be performed by a plurality of imagingdevices.

BACKGROUND ART

There are usage modes in which images captured by a plurality of imagingdevices are displayed on one screen.

For example, Patent Document 1 listed below discloses a technology fordisplaying an image of a relatively wide field of view and ahigh-resolution image of a relatively narrow field of view that areobtained by a first imaging optical system and a second imaging opticalsystem having different angles of view.

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2013-251783

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

As a situation in which the user performs imaging, there is a case wherethe user wishes to follow the target object while checking a wideperipheral field of view. In this case, it is convenient to display, onthe same screen, the image obtained by the first imaging devicecapturing a wide peripheral field of view and the image obtained by thesecond imaging device capturing the target object.

For such display, the technology disclosed in Patent Document 1 may beused, but, according to the disclosed technology, two images havingdifferent angles of view are displayed in divided screen regions.

As a result, movement of the line of sight of the user relative to thedisplay might become larger, or it might become difficult for the userto visually recognize a wide-angle image and a high-resolution image ofa moving object. Patent Document 1 also discloses displaying awide-angle image in a high-resolution image, but in this case, it mightbe difficult to visually recognize the object appearing in thewide-angle image.

Therefore, the present disclosure suggests a technology forappropriately displaying an image with a relatively wide imaging fieldof view captured by a first imaging device, and an image with arelatively narrow imaging field of view captured by a second imagingdevice.

Solutions to Problems

An image processing device according to the present technology includesan image combining unit that performs a combining process of disposing asecond image in a first image captured by a first imaging device, thesecond image being an image of the imaging field of view of a secondimaging device capable of imaging a partial field of view in the imagingfield of view of the first imaging device.

For example, a display image in which the second image having arelatively narrow field of view is superimposed on the first imagehaving a relatively wide field of view as an image of the imaging fieldof view of the first imaging device is generated.

The above image processing device according to the present technologymay further include: a range detection unit that detects the range ofthe imaging field of view of the second imaging device in the firstimage; and an imaging angle-of-view frame combining unit that combinesan imaging angle-of-view frame with the first image, the imagingangle-of-view frame indicating the range of the imaging field of viewdetected by the range detection unit.

That is, an image in which the range of the object being captured by thesecond imaging device is shown as the imaging angle-of-view frame in thefirst image is generated, and an image including the imagingangle-of-view frame is displayed.

In the above image processing device according to the presenttechnology, the image combining unit may set the layout position of thesecond image in the first image, in accordance with the position of theimaging angle-of-view frame.

When the second image is combined with the first image, the position ofthe combining is not set at a fixed position, but is set in accordancewith the position of the imaging angle-of-view frame.

In the above image processing device according to the presenttechnology, the image combining unit may set the layout position of thesecond image in the first image so as to follow the position of theimaging angle-of-view frame, with a predetermined distance relationshipbeing maintained.

The layout position of the second image to be combined with the firstimage is set for each frame so as to maintain a constant distance to theimaging angle-of-view frame, for example, and thus, the second imagefollows the imaging angle-of-view frame.

In the above image processing device according to the presenttechnology, the image combining unit may set the layout position of thesecond image in the first image so as to follow horizontal changes inthe position of the imaging angle-of-view frame while maintaining apredetermined horizontal distance.

For example, as the imaging angle-of-view frame moves in a horizontaldirection (transverse direction) in the first image, the second imagemoves while maintaining a certain horizontal distance.

In the above image processing device according to the presenttechnology, the image combining unit may set the layout position of thesecond image in the first image so as to follow vertical changes in theposition of the imaging angle-of-view frame while maintaining apredetermined vertical distance.

For example, as the imaging angle-of-view frame moves in a verticaldirection (upward/downward direction) in the first image, the secondimage moves while maintaining a certain vertical distance.

In the above image processing device according to the presenttechnology, the image combining unit may calculate the coordinate valuesof the layout position of the second image in the coordinate space ofthe first image by an arithmetic operation using the coordinate valuesof the imaging angle-of-view frame.

For example, the coordinates of the layout position of the second imagethat maintains a predetermined distance from the imaging angle-of-viewframe are calculated using the coordinate values of the center and fourcorners of the imaging angle-of-view frame.

In the above image processing device according to the presenttechnology, the image combining unit may perform segmentation setting todivide the first image into a plurality of areas, and set the layoutposition of the second image by a calculation formula selected inaccordance with the area in which the imaging angle-of-view frame islocated.

The layout position of the second image to be combined with the firstimage is set so as to follow the imaging angle-of-view frame whoseposition changes in the first image. In this case, segmentation settingis performed to divide the first image into a plurality of areas, thecalculation formula corresponding to the area in which the imagingangle-of-view frame (the center of the imaging angle-of-view frame, forexample) is located is selected, and the position of the second image isset.

In the above image processing device according to the presenttechnology, the plurality of areas may include a buffer area thatmaintains a state in which the calculation formula is selected.

In a case where the first image is divided into a plurality of areas,not the calculation formula corresponding to each area is prepared forall the areas, but a buffer area in which the calculation formulapreviously used is also set.

In the above image processing device according to the presenttechnology, the image combining unit may set the layout position of thesecond image in the first image in accordance with the position of theimaging angle-of-view frame, and, in a case where the set layoutposition is such that all or part of the second image is located outsidethe range of the first image, the image combining unit may again set thelayout position of the second image to change the direction of disposingthe second image relative to the imaging angle-of-view frame.

For example, when the layout position of the second image is set so thatthe direction of disposing the second image relative to the imagingangle-of-view frame is a direction toward the right side, in a casewhere the second image has a portion outside the first image, resettingis performed so that the direction of disposing the second imagerelative to the imaging angle-of-view frame becomes a direction towardthe left side.

Alternatively, when the layout position of the second image is set sothat the direction of disposing the second image relative to the imagingangle-of-view frame is a direction toward the upper side, in a casewhere the second image has a portion outside the first image, resettingis performed so that the direction of disposing the second imagerelative to the imaging angle-of-view frame becomes a direction towardthe lower side.

In the above image processing device according to the presenttechnology, the image combining unit may set the layout position of thesecond image in the first image in accordance with the position of theimaging angle-of-view frame, and, in a case where the set layoutposition is such that all or part of the second image is located outsidethe range of the first image, the image combining unit may again set thelayout position of the second image so as to cause the layout positionof the second image to shift toward the range of the first image.

For example, in a case where the layout position of the second imagethat is set relative to the imaging angle-of-view frame is such aposition as to have a portion outside the first image, the direction ofdisposing the second image is made to shift in a direction of reducingthe outside portion.

In the above image processing device according to the presenttechnology, the image combining unit may set the layout position of thesecond image in the first image in accordance with the position of theimaging angle-of-view frame, and, in a case where the set layoutposition is such that all or part of the second image is located outsidethe range of the first image, the combining process may not be performedon the second image.

That is, in a case where the layout position of the second image is aposition having a portion outside the first image, the second image isnot displayed.

The above image processing device according to the present technologymay further include an output control unit that performs control tooutput the second image as a display image, in a case where the rangedetection unit has detected the second imaging device in the firstimage.

That is, this is a state in which the second imaging device is locatedin front of the first imaging device, and the back surface of the secondimaging device is captured in the first image. In this case, the secondimage is not combined with the first image, and only the second image isdisplayed to the user.

The output control unit may correspond to the image combining unit. Thatis, this is a case where the image combining unit performs control tooutput the second image as a display image. Also, there may be a casewhere the second image of the second imaging device is output withoutpassing through the image combining unit. In that case, a function asthe output control unit should be executed by a unit other than theimage combining unit.

In the above image processing device according to the presenttechnology, in a case where the range detection unit fails to detect therange of the imaging field of view of the second imaging device in thefirst image, the image combining unit may set the layout position of thesecond image at a preset fixed position in the first image.

For example, in a case where the imaging visual field directions of thefirst imaging device and the second imaging device are completelydifferent, the range of the imaging field of view of the second imagingdevice does not exist in the first image. In such a case, the secondimage is combined and displayed at a specific position in the firstimage.

The above image processing device according to the present technologymay further include an output control unit that performs control tooutput the first image as a display image, in a case where the rangedetection unit fails to detect the range of the imaging field of view ofthe second imaging device in the first image.

For example, in a case where the imaging visual field directions of thefirst imaging device and the second imaging device are completelydifferent, and the range of the imaging field of view of the secondimaging device does not exist in the first image, the second image isnot displayed, and the first image is displayed.

The output control unit may also correspond to the image combining unitin this case. That is, this is a case where the image combining unitperforms control to output the first image as a display image. Also,there may be a case where the first image of the first imaging device isoutput without passing through the image combining unit. In that case, afunction as the output control unit should be executed by a unit otherthan the image combining unit.

In the above image processing device according to the presenttechnology, the image combining unit may determine whether or not tocombine the second image with the first image, in accordance with thesize of the imaging angle-of-view frame.

The size of the imaging angle-of-view frame changes with the angle ofview (zooming state) of the second imaging device. Particularly, whenimaging is performed at a wide angle, the angle of view of the secondimaging device approaches the angle of view of the first imaging device,and the range occupied by the imaging angle-of-view frame becomes widerin the first image. In such a case, the second image is not displayed.

In the above image processing device according to the presenttechnology, the imaging angle-of-view frame combining unit may combinethe imaging angle-of-view frame that is non-rectangular in shape withthe first image.

For example, the imaging angle-of-view frame is displayed in the shapecorresponding to the difference in the visual field direction betweenthe first and second imaging devices.

The above image processing device according to the present technologymay further include: a range detection unit that detects the range of afocus frame of the second imaging device in the first image; and a focusframe combining unit that combines the focus frame with the first image,the focus frame indicating the focusing region detected by the rangedetection unit. The image combining unit may set the layout position ofthe second image in the first image in accordance with the position ofthe focus frame.

That is, the focus frame of the second imaging device is displayed inthe first image. The coordinates of the layout position of the secondimage that maintains a predetermined distance from the imagingangle-of-view frame are then calculated, on the basis of the coordinatevalues of the focus frame, for example.

An image processing method according to the present technology is animage processing method by which an image processing device performs acombining process of disposing a second image in a first image capturedby a first imaging device, the second image being an image of theimaging field of view of a second imaging device capable of imaging apartial field of view in the imaging field of view of the first imagingdevice.

By this method, the first image is displayed in the second image havinga relatively wide imaging field of view.

A program according to the present technology is a program for causingan image processing device to perform processes by the above imageprocessing method. With this program, an image processing deviceaccording to the present technology can be easily obtained.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an example configuration according to anembodiment of the present technology.

FIG. 2 is an explanatory diagram of an example and a usage state of twoimaging devices according to an embodiment.

FIG. 3 is an explanatory diagram of an example of two imaging devicesaccording to an embodiment.

FIG. 4 is an explanatory view of a peripheral field image, an imagingangle-of-view frame, and an enlarged image displayed according to anembodiment.

FIG. 5 is a block diagram of an imaging device according to anembodiment.

FIG. 6 is a flowchart of an example process according to first to fifthembodiments.

FIG. 7 is a flowchart of an enlarged image layout coordinate calculationprocess according to the first embodiment.

FIG. 8 is an explanatory diagram of the x-y coordinate system to be usedin various calculations according to an embodiment.

FIG. 9 is an explanatory diagram of the coordinate values to be used invarious calculations according to an embodiment.

FIG. 10 is an explanatory diagram of area segmentation and enlargedimage layout position setting according to the first embodiment.

FIG. 11 is an explanatory diagram of display images in an eye proximitystate according to an embodiment.

FIG. 12 is an explanatory diagram of a state of specific positiondisplay or non-display of an enlarged image according to an embodiment.

FIG. 13 is an explanatory diagram of non-display of an enlarged imagedepending on the size of an imaging angle-of-view frame according to anembodiment.

FIG. 14 is an explanatory diagram of a process to be performed when anenlarged image has an outside portion according to an embodiment.

FIG. 15 is an explanatory diagram of a focus frame according to a secondembodiment.

FIG. 16 is a flowchart of an enlarged image layout coordinatecalculation process according to the second embodiment.

FIG. 17 is a flowchart of an enlarged image layout coordinatecalculation process according to a third embodiment.

FIG. 18 is an explanatory diagram of enlarged image position settingaccording to a fourth embodiment.

FIG. 19 is an explanatory diagram of enlarged image position settingaccording to a fifth embodiment.

FIG. 20 is a flowchart of an enlarged image layout coordinatecalculation process according to the fifth embodiment.

FIG. 21 is an explanatory diagram of a modification of area segmentationsetting according to the fifth embodiment.

FIG. 22 is an explanatory diagram of enlarged image position settingaccording to a sixth embodiment.

FIG. 23 is an explanatory diagram of the layout relationship between animaging angle-of-view frame and an enlarged image according to the sixthembodiment.

FIG. 24 is a flowchart of example processes according to the sixth toseventh embodiments.

FIG. 25 is a flowchart of a layout coordinate recalculation processaccording to the sixth embodiment.

FIG. 26 is an explanatory diagram of right-left switching to beperformed when an enlarged image has an outside portion according to thesixth embodiment.

FIG. 27 is an explanatory diagram of horizontal shifts to be performedwhen an enlarged image has an outside portion according to the sixthembodiment.

FIG. 28 is an explanatory diagram of enlarged image position settingaccording to the seventh embodiment.

FIG. 29 is an explanatory diagram of the layout relationship between animaging angle-of-view frame and an enlarged image according to theseventh embodiment.

FIG. 30 is a flowchart of a layout coordinate recalculation processaccording to the seventh embodiment.

FIG. 31 is an explanatory diagram of vertical shifts to be performedwhen an enlarged image has an outside portion according to the seventhembodiment.

FIG. 32 is an explanatory view of a display mode of an imagingangle-of-view frame according to an embodiment.

MODE FOR CARRYING OUT THE INVENTION

Embodiments will be described below in the following order.

<1. Configuration of an Image Processing Device>

<2. First Embodiment>

<3. Second Embodiment>

<4. Third Embodiment>

<5. Fourth Embodiment>

<6. Fifth Embodiment>

<7. Sixth Embodiment>

<8. Seventh Embodiment>

<9. Display of an Imaging Angle-of-View Frame>

<10. Summary and Modifications>

1. Configuration of an Image Processing Device

FIG. 1 shows an example configuration of an image processing device 3 asan embodiment. FIG. 1 shows imaging devices 1A and 1B, a display device2, and an image processing device 3.

The imaging devices 1A and 1B represent two separate devices capable ofcapturing images, for example. The imaging devices 1A and 1B are devicescapable of capturing moving images and still images, and recording theimages. FIGS. 2 and 3 show examples of the imaging devices 1A and 1B.

FIG. 2A shows, as an example, an imaging device 1A capable ofgoggle-like display and imaging, and an imaging device 1B as a handheldconventional digital camera.

For example, the user wears the goggle-like imaging device 1A on thehead. The imaging device 1A is a device that includes an imaging unit32, for example, and is capable of capturing an image in a relativelywide imaging field of view and displaying the captured image on adisplay unit (not shown) in the goggles.

The imaging device 1B is a device capable of capturing a still image ora moving image and recording the captured still image or moving image ona recording medium such as a memory card.

The imaging device 1A and the imaging device 1B can communicate imagedata and other data to each other by cable communication or wirelesscommunication.

The imaging devices 1A and 1B are not limited to the combination shownin FIG. 2A, and combinations of various imaging devices are possible.

For example, FIG. 3A shows a case where both the imaging devices 1A and1B are handheld digital cameras.

FIG. 3B shows an example in which the imaging device 1A is a 360-degreecamera that performs imaging in all directions with a fisheye lens, forexample, and the imaging device 1B is a portable terminal such as asmartphone having an imaging function.

FIG. 3C shows an example in which the imaging device 1A is asurveillance camera and the imaging device 1B is a video camera.

As described above, there are various combinations of the imagingdevices 1A and 1B, and various combinations are possible other than theabove examples.

In this embodiment, however, at least the imaging device 1B is a devicecapable of imaging a partial field of view in the imaging field of viewof the imaging device 1A. For example, in the case shown in FIG. 2A, thegoggle-type imaging device 1A can image the peripheral field of viewwith a wide imaging field of view (angle of view), so that the user canvisually recognize a view in the peripheral field of view on an internaldisplay unit. After that, it is possible to capture an image of thetarget object existing in the peripheral field of view with the imagingdevice 1B.

Although the two imaging devices 1A and 1B as described above are shownin FIG. 1 , the image processing device 3 and the display device 2 maybe devices independent of the imaging devices 1A and 1B, or both of themmay be included in the imaging device 1A or the imaging device 1B.

For example, in a case where the goggle-type imaging device 1A and theimaging device 1B that is a handheld digital camera are being used asshown in FIG. 2A, the image processing device 3 and the display device 2in FIG. 1 may be included in the imaging device 1A.

It is of course conceivable to form a configuration in which the imageprocessing device 3 and the display device 2 are included in the imagingdevice 1B, the image processing device 3 is included in the imagingdevice 1B while the display device 2 is included in the imaging device1A, or conversely, the image processing device 3 is included in theimaging device 1A while the display device 2 is included in the imagingdevice 1B.

In the description below, the imaging devices 1A and 1B will bedescribed as separate devices. However, in a case where two imagingsystems (lens systems, image sensors, and the like) are included in oneimaging device and can perform imaging of two systems having adifference in the imaging field of view, the two imaging systems can beapplied to the imaging devices 1A and 1B.

The image processing device 3 has processing functions that are thefunctions as a range detection unit 10, an imaging angle-of-view framecombining unit 11, an image combining unit 12, and a display controlunit 13.

For example, the image processing device 3 is designed as a videoprocessor including an arithmetic device such as a microcomputer, andprocessing functions as the range detection unit 10, the imagingangle-of-view frame combining unit 11, the image combining unit 12, andthe display control unit 13 are realized in an arithmetic unit such as aCPU according to a software program.

Image data captured by the imaging devices 1A and 1B and various kindsof additional information are supplied to the image processing device 3.

The image data to be supplied from the imaging devices 1A and 1B to theimage processing device 3 is image data for the user who performsimaging to check the object range, for example. Therefore, the imagedata may be image data with a relatively low resolution as a so-calledthrough-lens image, row image data, image data obtained by developing arow image, or the like, or may be data with a relatively highresolution.

Conceivable examples of the additional information to be supplied fromthe imaging devices 1A and 1B to the image processing device 3 includeinformation about the focus control position and the zoom lens controlposition (or the angle of view) during imaging, exposure information,detection information about various sensors mounted on the imagingdevices 1A and 1B, such as an orientation sensor, a distance measuringsensor, an angular velocity sensor, an acceleration sensor, anilluminance sensor, and a location sensor, and information about theuser's operations on the imaging device 1A or the imaging device 1B, forexample.

The range detection unit 10 in the image processing device 3 performs aprocess of detecting the range of the imaging field of view of theimaging device 1B in a first image captured by the imaging device 1A.For example, in a case where the peripheral field is imaged by thegoggle-type imaging device 1A as shown in FIG. 2A, the process is aprocess of detecting the range of the field of view being imaged by theimaging device 1B in the image of the peripheral field.

For example, FIG. 4A shows an example image captured as a peripheralfield image 20 by the imaging device 1A. In this example, the imagingdevice 1B images the range of the field of view indicated by an imagingangle-of-view frame 21. The range detection unit 10 performs a processof detecting the range indicated by the imaging angle-of-view frame 21in this manner in the peripheral field image 20.

The position of the imaging field of view of the imaging device 1B inthe peripheral field image 20 to be detected by the range detection unit10 changes with the respective field-of-view directions of the imagingdevice 1A and the imaging device 1B. Also, the size of the imaging fieldof view of the imaging device 1B in the peripheral field image 20 to bedetected by the range detection unit 10 changes with a difference inangle of view between the imaging device 1A and the imaging device 1B.For example, the size changes with the zooming state of the imagingdevice 1B.

Accordingly, for each frame, for example, the range detection unit 10detects the range of the imaging field of view of the imaging device 1Bin the first image captured by the imaging device 1A.

For example, the range detection unit 10 can acquire the image data ofthe peripheral field and the additional information from the imagingdevice 1A, and also acquire the image data of the field of view beingimaged and the additional information from the imaging device 1B. Therange detection unit 10 detects the difference in imaging conditionsbetween the imaging device 1A and the imaging device 1B, and detects theimaging field of view of the imaging device 1B relative to theperipheral field of view. The difference in imaging conditions may be animaging direction difference or an angle-of-view difference (anenlargement factor), for example. Once an imaging direction differenceand an angle-of-view difference are acquired, the imaging field of viewof the imaging device 1B in the peripheral field of view of the imagingdevice 1A can be detected by calculation.

Also, the range detection unit 10 may calculate the imaging field ofview of the imaging device 1B relative to the peripheral field of viewby image analysis. For example, it is also possible to detect theimaging field of view of the imaging device 1B in the peripheral fieldof view of the imaging device 1A, by performing processes such asso-called block matching, feature point extraction, and feature pointmatching on the object in the image data from the imaging device 1B andthe image data of the peripheral field of view from the imaging device1A, and detecting the matching portions.

The imaging angle-of-view frame combining unit 11 performs a process ofcombining the imaging angle-of-view frame 21 indicating the range of theimaging field of view detected by the range detection unit 10. That is,a frame image as the imaging angle-of-view frame 21 indicated by adashed line in FIG. 4A is combined with the peripheral field image 20.

As a result, the user can visually recognize the peripheral field image20 to which the imaging angle-of-view frame 21 is added, and check whichportion is being imaged by the imaging device 1B.

Since the position and the size of the imaging field of view of theimaging device 1B in the peripheral field image 20 to be detected by therange detection unit 10 change with each frame as described above, theimaging angle-of-view frame combining unit 11 performs, for each frame,the process of combining the image of the imaging angle-of-view frame 21with the detected range.

The image combining unit 12 performs a combining process of disposing asecond image, which is the image of the imaging field of view of theimaging device 1B, so as to overlap the first image (the peripheralfield image 20) captured by the imaging device 1A. An example of thesecond image is an enlarged image 22 shown in FIG. 4A. The image contentcaptured by the imaging device 1B and displayed as the enlarged image 22is the image content obtained by enlarging the image in the imagingangle-of-view frame 21.

As a result, the user can visually recognize the image captured by theimaging device 1B in detail while viewing the peripheral field image 20captured by the imaging device 1A.

The size of the enlarged image 22 is only required to be a fixed size,but may be variably set by a user operation, for example.

Note that, since the size of the imaging angle-of-view frame 21 dependson the angle-of-view difference between the imaging devices 1A and 1B,there is a possibility that the size of the imaging angle-of-view frame21 may be equal to that of the enlarged image 22 or larger than that ofthe enlarged image 22, and the image content might not be enlarged insome cases. However, the “enlarged image 22” is originally displayed forthe purpose of displaying an enlarged image, and therefore, is calledthat name, for ease of explanation. Note that, as will be describedlater, in this embodiment, the enlarged image 22 is not displayed in acase where the image in the enlarged image 22 is not in an “enlarged”state due to the size of the imaging angle-of-view frame 21. In thatsense, there will be no cases where the enlarged image 22 beingdisplayed is not an enlarged image.

The display control unit 13 performs control to cause the display device2 to display a combined image as shown in FIG. 4A, for example.

The user can visually recognize the image as shown in FIG. 4A on thedisplay device 2. Accordingly, in the peripheral field image 20, therange being imaged by the imaging device 1B can be checked with theimaging angle-of-view frame 21, and be further visually recognized indetail with the enlarged image 22.

Note that, in the drawing, the imaging angle-of-view frame 21 isindicated by a dashed line, and the frame of the enlarged image 22 isindicated and surrounded by a solid line. However, different displaymodes may be adopted to make it easier for the user to distinguish theimaging angle-of-view frame 21 and the enlarged image 22. As thedifferent display modes, various examples are conceivable, such asmaking the colors of the frames differ from each other, highlighting oneof the images, making one a color image while making the other amonochromatic image, or making the luminances of them differ from eachother.

In a case where the configuration shown in FIG. 1 is to be achieved withthe imaging devices 1A and 1B shown in FIG. 2A, it is conceivable thatthe image processing device 3 and the display device 2 may be includedin the goggle-type imaging device 1A, for example.

FIG. 5 shows an example configuration of the imaging device 1A in thatcase.

The imaging device 1A includes an imaging unit 32, an image signalprocessing unit 33, an image analysis unit 34, a control unit 35, anoperation unit 36, a sensor unit 37, a display control unit 38, adisplay unit 39, a storage unit 40, and an external input unit 41.

The imaging unit 32 includes an imaging optical system and an imagesensor for imaging. The image sensor is an imaging element such as acharge coupled device (CCD) sensor or a complementary metal oxidesemiconductor (CMOS) sensor, for example, receives light entering fromthe object through the imaging optical system, converts the light intoan electrical signal, and outputs the electrical signal. In the imagesensor, a correlated double sampling (CDS) process, an automatic gaincontrol (AGC) process, or the like, for example, is performed on anelectrical signal obtained by photoelectrically converting receivedlight, and an analog/digital (A/D) conversion process is furtherperformed on the electrical signal. Image data as digital data is thenoutput to the image signal processing unit 33 in the subsequent stage.

The image signal processing unit 33 is formed as an image processor,such as a digital signal processor (DSP), for example. The image signalprocessing unit 33 performs various kinds of processing on the imagedata input from the imaging unit 32.

For example, in a case where an image signal is assumed to be a normalvisible light image, the image signal processing unit 33 performsprocesses such as a clamping process of clamping the black levels of red(R), green (G), and blue (B) to a predetermined level, a correctionprocess among the color channels of R, G, and B, a demosaicing processof causing the image data for each pixel to include all the colorcomponents of R, G, and B, and a process of generating (separating) aluminance (Y) signal and a color (C) signal.

In some cases, the image signal processing unit 33 further performs anecessary resolution conversion process, such as resolution conversionfor recording, communication outputs, or monitor images, for example, onthe image signal subjected to the various kinds of signal processing.

There also are cases where the image signal processing unit 33 performsa compression processing for recording or communication, for example, enencoding process, or the like on the image data subjected to theresolution conversion.

The image analysis unit 34 performs an image analysis process on eachframe (or intermittent frame) of the image signal subjected topredetermined processing by the image signal processing unit 33. Forexample, by a method such as pattern matching, it is possible todetermine the type and an operating state of a body as the object,determine the region and the attribute of the object to measure thefeature amount, or perform measurement based on the determination.

The information determined by the image analysis unit 34 is supplied tothe control unit 35, and is used in a series of processes formeasurement.

Note that the image analysis unit 34 may be designed as an artificialintelligence (AI) engine, perform an image recognition process based onmachine learning, deep learning, or the like, and be capable ofdetermining a body or recognizing feature points, for example.

The control unit 35 is formed with a microcomputer (an arithmeticprocessing device) that includes a central processing unit (CPU), a readonly memory (ROM), a random access memory (RAM), and a flash memory.

The CPU comprehensively controls the entire imaging device 1A byexecuting a program stored in the ROM, the flash memory, or the like.

The RAM is used as a work area for various kinds of data processing bythe CPU, to temporarily store data, programs, and the like.

The ROM and the flash memory (nonvolatile memory) are used to store anoperating system (OS) for the CPU to control each component, contentfiles such as image files, application programs for various kinds ofoperations, firmware, and the like.

Such a control unit 35 performs control related to imaging operationssuch as the shutter speed, exposure adjustment, and the frame rate inthe imaging unit 32, parameter control in various kinds of signalprocessing in the image signal processing unit 33, and control on theanalysis process being performed by the image analysis unit 34. Thecontrol unit 35 also performs a setting process, imaging operationcontrol, display operation control, and the like, in accordance with theuser's operations.

Note that the control unit 35 may include the functions of the imageanalysis unit 34.

The operation unit 36 may be operating components such as keys,switches, or dials provided on the device housing, or may be a touchpanel or the like. Through this operation unit 36, the user may performpower-on/off operations, various setting operations, target value inputoperations, program activation, and the like, for example. The operationunit 36 transmits a signal corresponding to an input operation, to thecontrol unit 35.

The sensor unit 37 comprehensively represents various sensors providedas necessary. For example, there may be various sensors such as a voicesensor, a location sensor, an illuminance sensor, a contact sensor, atemperature sensor, a distance measuring sensor, an acceleration sensor,an angular velocity sensor, an atmospheric pressure sensor, an altitudesensor, and a pressure sensor.

The display unit 39 is a display unit that performs various kinds ofdisplay for the user (the person conducting the imaging or the like),and is formed with a display device such as a liquid crystal display(LCD) or an organic electro-luminescence (EL) display provided in thegoggles in the goggle-type imaging device 1A, for example. The user canview an image of the object on the display unit 39 in front of the eyesby wearing the imaging device 1A.

The display control unit 38 performs a process of causing the displayunit 39 to perform a display operation. For example, the display controlunit includes a character generator, a display driver, and the like, forexample, and, on the basis of control by the control unit 35, causes thedisplay unit 39 to perform various kinds of display, such as the imagedisplay shown in FIG. 4A, for example.

Further, the display control unit 38 may cause the display unit 39 toreproduce and display a still image or a moving image recorded on arecording medium.

On the basis of an instruction of the control unit 35, the displaycontrol unit 38 may also cause the display unit 39 to display variousoperation menus, icons, messages, and the like, which are displays asgraphical user interfaces (GUI) on the screen.

The storage unit 40 is formed with a nonvolatile memory, for example,and stores image files of still image data, moving image data, and thelike captured by the imaging unit 32, for example, attribute informationabout the image files, thumbnail images, and the like.

The storage unit 40 can take various forms in practice. For example, thestorage unit 40 may be a flash memory included in the imaging device 1A,or may be formed with a memory card (for example, a portable flashmemory) that can be detachably attached to the imaging device 1A and acard recording/reproducing unit that makes recording/reproducing accessto the memory card. Alternatively, as a component to be included in theimaging device 1A, the storage unit 40 may be formed as a hard diskdrive (HDD) or the like.

The external input unit 41 receives inputs of information from anexternal device by cable communication or wireless communication. Inthis case, it is conceivable that the external input unit is designed asa communication unit for receiving inputs of image data and additionalinformation from the imaging device 1B.

In the above configuration, the image processing device 3 in FIG. 1 isformed as part of the functions of the image signal processing unit 33,the image analysis unit 34, the control unit 35, and the display controlunit 38, for example. Also, the display device 2 in FIG. 1 is formedwith the display unit 39.

Note that, although the configuration of the imaging device 1A has beendescribed with reference to FIG. 5 , the imaging device 1B also has aconfiguration substantially equal to that shown in FIG. 5 , as long asit functions as a conventional camera.

2. First Embodiment

In the description below, various examples of processes to be performedby the image processing device 3 according to embodiments will bedescribed.

In this embodiment, the image processing device 3 (the imaging device 1Aincluding the image processing device 3, for example) performs displayas shown in FIG. 4A on the display unit 39, so that the user can viewthe imaging angle-of-view frame 21 and the enlarged image 22, as well asthe peripheral field image 20.

The enlarged image 22 may be displayed to be superimposed on a fixedposition within the peripheral field image 20, for example.Particularly, as the enlarged image 22 is displayed within theperipheral field image 20, the user can visually recognize both theentire peripheral field image 20 and the enlarged image 22 withoutmoving the line of sight greatly, and it is convenient for the user tocheck both images, which are the periphery and the target object. As theimaging angle-of-view frame 21 is also within the peripheral field image20, it is of course easy to visually recognize the imaging angle-of-viewframe 21, and the user can easily recognize the positional state of theobject captured in the enlarged image 22.

However, as described above, the position and the size of the imagingangle-of-view frame 21 change. For this reason, the imagingangle-of-view frame 21 may have a position or a size to overlap theenlarged image 22 in some cases. In such a case, either the imagingangle-of-view frame 21 or the enlarged image 22 becomes difficult tosee, or visibility deteriorates due to the overlap.

Therefore, in the embodiment, the layout position of the enlarged image22 is made variable depending on the imaging angle-of-view frame 21, sothat an easy-to-view display state is maintained.

For example, as shown in FIG. 4A, the enlarged image 22 is disposed at apredetermined distance on the right side of the imaging angle-of-viewframe 21. However, in a case where the imaging angle-of-view frame 21 isat a position on the right side in the peripheral field image 20, theenlarged image 22 is disposed at a predetermined distance on the leftside, as shown in FIG. 4B, for example.

With this arrangement, for example, both the imaging angle-of-view frame21 and the enlarged image 22 become easily visible, regardless of theposition and the size of the imaging angle-of-view frame 21.

A specific example as the first embodiment is now described withreference to FIGS. 6 and 7 . FIGS. 6 and 7 show processes to beperformed by the image processing device 3 having the functions of therange detection unit 10, the imaging angle-of-view frame combining unit11, the image combining unit 12, and the display control unit 13.

Particularly, this example of processes involves processes to beperformed in a case where, while the peripheral field image 20 beingcaptured by the imaging device 1A is displayed on the display device 2,the imaging angle-of-view frame 21 and the enlarged image 22 of theimage (a through-lens image or the like) being captured by the imagingdevice 1B are superimposed and displayed on the peripheral field image20.

The processes shown in FIG. 6 are performed for each one frame of imagedata captured by the imaging device 1B, for example. Note that theprocesses may be performed for each intermittent frame.

In step S101 in FIG. 6 , the image processing device 3 performs visualfield range detection with the functions of the range detection unit 10.This process is a process of detecting which range in the peripheralfield image 20 being currently captured by the imaging device 1Acorresponds to the image of the frame being currently captured by theimaging device 1B. That is, the range in which the imaging angle-of-viewframe 21 is to be displayed is detected.

As described above, the image processing device 3 detects the differencein imaging conditions between the imaging device 1A and the imagingdevice 1B, and detects the imaging field of view of the imaging device1B relative to the peripheral field of view. Alternatively, the imagingfield of view of the imaging device 1B relative to the peripheral fieldof view is detected by image analysis.

Note that the field of view as the peripheral field image 20 variesdepending on the imaging direction and the zooming state of the imagingdevice 1A, and the field of view of the image being captured by theimaging device 1B also varies depending on the imaging direction and thezooming state. Therefore, in the visual field range detection in stepS101, an image range that matches the captured image in the currentframe of the imaging device 1B is detected in the captured image in thecurrent frame of the imaging device 1A.

In step S103, at the image processing device 3, the process branchesdepending on whether or not the imaging field of view of the imagingdevice 1B is successfully detected in the peripheral field image 20. Forexample, in a case where the imaging device 1B blocks the field of viewin front of the lens of the imaging device 1A as shown in FIG. 2B, or ina case where the imaging directions of the imaging devices 1A and 1B aregreatly different from each other as shown in FIG. 2C, the imaging fieldof view of the imaging device 1B may not be included in the peripheralfield image 20 in some cases.

On the other hand, in a case where the imaging devices 1A and 1B havesubstantially the same imaging directions as shown in FIG. 2A, or in acase where the imaging directions are slightly different from eachother, the imaging field of view of the imaging device 1B is included inthe peripheral field image 20, and the detection in step S101 ispossible.

First, a case where the imaging field of view of the imaging device 1Bis successfully detected in the peripheral field image 20 is described.

In that case, the image processing device 3 proceeds from step S102 tostep S103, and determines the size of the imaging angle-of-view frame 21to be displayed. That is, it is the size of the range of the imagingfield of view of the imaging device 1B in the peripheral field image 20detected in step S101, and is the area thereof, for example.

In step S104, the process branches depending on whether or not the sizeof the imaging angle-of-view frame 21 is determined to be large. In thiscase, for example, it is determined whether or not the size of theimaging angle-of-view frame 21 is larger than the size of the enlargedimage 22. Alternatively, it is determined whether or not the size of theimaging angle-of-view frame 21 is larger than a predetermined value. Theprocesses in steps S103 and S104, and in a case where the size isdetermined to be large will be described later.

For example, if the difference in angle of view between the imagingdevices 1A and 1B is large, and the size is determined not to be largein step S104, the image processing device 3 moves on to step S105, andperforms enlarged image layout coordinate calculation.

The process of this enlarged image layout coordinate calculation isshown in FIG. 7 .

Here, prior to explanation with reference to FIG. 7 , the coordinatesystem to be used for the calculation is described with reference toFIG. 8 .

FIG. 8 shows a coordinate system using the x-axis and the y-axis, withthe central coordinates of the peripheral field image 20 being theorigin (0, 0). In the coordinate system of the peripheral field image20, the positions of the imaging angle-of-view frame 21 and the enlargedimage 22 are expressed by coordinates.

Specifically, as shown in FIG. 9 , the coordinates and the size of theimaging angle-of-view frame 21 are expressed as follows.

Central coordinates: (x0, y0)

Upper left corner coordinates: (xul, yul)

Upper right corner coordinates: (xur, yur)

Lower left corner coordinates: (xdl, ydl)

Lower right corner coordinates: (xdr, ydr)

Vertical size: H

Horizontal size: W

Likewise, the coordinates and the size of enlarged image 22 areexpressed as follows.

Central coordinates: (X0, Y0)

Upper left corner coordinates: (Xul, Yul)

Upper right corner coordinates: (Xur, Yur)

Lower left corner coordinates: (Xdl, Ydl)

Lower right corner coordinates: (Xdr, Ydr)

Vertical size: Hz

Horizontal size: Wz

The image processing device 3 acquires imaging angle-of-view frameinformation in step S150 in FIG. 7 in the enlarged image layoutcoordinate calculation. That is, the respective coordinate values of thecenter and the four corners of the imaging angle-of-view frame 21 shownin FIG. 9 are acquired on the basis of information about the field ofview obtained in step S101 in FIG. 6 .

In step S151, the image processing device 3 searches for the center ofthe imaging angle-of-view frame 21, which is the area in which thecentral coordinates (x0, y0) are located.

Here, the image processing device 3 sets a plurality of divided areas inthe peripheral field image 20. As an example, FIG. 10A shows areas AR1,AR2, and AR3 obtained by horizontally dividing the peripheral fieldimage 20 into three at boundary lines BD1 and BD2.

Note that the peripheral field image may be divided so that therespective areas have the same size, but FIG. 10A shows an example inwhich the central area AR2 is narrower than the others.

In the area search in step S151 of FIG. 7 , it is determined in whicharea the central coordinates (x0, y0) are located among the areas AR1,AR2, and AR3. Since the image processing device 3 grasps the range orthe boundary of each of the areas AR1, AR2, and AR3 by the coordinatevalues, this process can be performed by a simple comparison calculationof the coordinate values.

In step S152, at the image processing device 3 the process branchesdepending on whether or not the central coordinates (x0, y0) are locatedin the area AR3.

In a case where the central coordinates (x0, y0) are located in the areaAR1 or the area AR2, the image processing device 3 moves on to stepS153, and calculates the layout coordinates of the enlarged image 22using the calculation formulas set for the areas AR1 and AR2.

In a case where the central coordinates (x0, y0) are located in the areaAR3, on the other hand, the image processing device 3 moves on to stepS154, and calculates the layout coordinates of the enlarged image 22using the calculation formulas set for the area AR3.

For example, FIG. 10B shows a case where the central coordinates (x0,y0) of the imaging angle-of-view frame 21 are located in the area AR2,and FIG. 10C shows a case where the central coordinates (x0, y0) of theimaging angle-of-view frame 21 are located in the area AR1. In eithercase, the enlarged image 22 is disposed, with a predetermined horizontaloffset amount offset_x being provided on the right side of the imagingangle-of-view frame 21.

On the other hand, FIG. 10D shows a case where the central coordinates(x0, y0) of the imaging angle-of-view frame 21 are located in the areaAR3. In either case, the enlarged image 22 is disposed, also with thehorizontal offset amount offset_x being provided on the left side of theimaging angle-of-view frame 21.

In this manner, the layout position of the enlarged image 22 is madedifferent between the right side and the left side, depending on thearea in which the central coordinates (x0, y0) of the imagingangle-of-view frame 21 are located. The enlarged image 22 is thendisposed, with the constant horizontal offset amount offset_x beingprovided relative to the imaging angle-of-view frame 21.

Various examples are conceivable as the layout coordinate calculationfor disposing the enlarged image 22 in such a manner, but first, examplecalculations based on the end sides of the imaging angle-of-view frame21 are described.

-   -   Calculation Formulas for the Areas AR1 and AR2

In a case where the central coordinates (x0, y0) of the imagingangle-of-view frame 21 are located in the area AR1 or AR2, the imageprocessing device 3 calculates the coordinates of the four corners ofthe enlarged image 22 in the process in step S153 as shown below.

(Xul,Yul)=(xr+offset_x,y0+Hz/2)

(Xur,Yur)=(xr+offset_x+Wz,y0+Hz/2)

(Xdl,Ydl)=(xr+offset_x,y0−Hz/2)

(Xdr,Ydr)=(xr+offset_x+Wz,y0−Hz/2)

Here, “xr” represents the x-coordinate value of the right end side ofimaging angle-of-view frame 21 (see FIGS. 10B and 10C). That is,xr=xur=xdr.

Calculation Formulas for the Area AR3

In a case where the central coordinates (x0, y0) of the imagingangle-of-view frame 21 are located in the area AR3, the image processingdevice 3 calculates the coordinates of the four corners of the enlargedimage 22 in the process in step S154 as shown below.

(Xul,Yul)=(xl−offset_x−Wz,y0+Hz/2)

(Xur,Yur)=(xl−offset_x,y0+Hz/2)

(Xdl,Ydl)=(xl−offset_x−Wz,y0−Hz/2)

(Xdr,Ydr)=(xl−offset_x,y0−Hz/2)

Further, “xl” represents the x-coordinate value of the left end side ofimaging angle-of-view frame 21 (see of FIG. 10D). That is, xl=xul=xdl.

Instead of the above, the layout coordinate calculation for disposingthe enlarged image 22 can also be performed by example calculationsbased on the central coordinates (x0, y0) of the imaging angle-of-viewframe 21 as shown below.

Calculation Formulas for the Areas AR1 and AR2

In a case where the central coordinates (x0, y0) of the imagingangle-of-view frame 21 are located in the area AR1 or AR2, the imageprocessing device 3 calculates the central coordinates (X0, Y0) of theenlarged image 22 in the process in step S153 as shown below.

(X0,Y0)=(x0+W/2+offset_x+Wz/2,y0)

The coordinate values of the four corners of the enlarged image 22 arethen calculated as follows.

(Xul,Yul)=(X0−Wz/2,Y0+Hz/2)

(Xur,Yur)=(X0+Wz/2,Y0+Hz/2)

(Xdl,Ydl)=(X0−Wz/2,Y0−Hz/2)

(Xdr,Ydr)=(X0+Wz/2,Y0−Hz/2)

Calculation Formulas for the Area AR3

In a case where the central coordinates (x0, y0) of the imagingangle-of-view frame 21 are located in the area AR3, the image processingdevice 3 calculates the central coordinates (X0, Y0) of the enlargedimage 22 in the process in step S154 as shown below.

(X0,Y0)=(x0−W/2−offset_x−Wz/2,y0)

The coordinate values (Xul, Yul), (Xur, Yur), (Xdl, Ydl), and (Xdr, Ydr)of the four corners of the enlarged image 22 are then calculated in amanner similar to the above.

For example, after the layout position of the enlarged image 22 iscalculated in step S153 or step S154 in FIG. 7 by the calculation basedon the end sides or the center of the imaging angle-of-view frame 21 asdescribed above, the image processing device 3 moves on to step S106 inFIG. 6 .

In step S106, the image processing device 3 checks whether or not thereis a portion outside the peripheral field image 20, as the layout stateof the enlarged image 22.

Processes related to the check of an outside portion will be describedlater.

In a case where it is determined that the enlarged image 22 does nothave any portion protruding from the peripheral field image 20, theimage processing device 3 moves on to step S107, and control display ofthe imaging angle-of-view frame 21 and the enlarged image 22 in theperipheral field image 20.

As a result, the imaging angle-of-view frame 21 and the enlarged image22 are displayed in the peripheral field image 20, with the displaydevice 2 in the layout state shown in FIG. 10B, 10C, or 10D.

Particularly, since the enlarged image 22 is displayed with thepredetermined horizontal offset amount offset_x in the directiondepending on the areas of the imaging angle-of-view frame 21, theimaging angle-of-view frame 21 and the enlarged image 22 do not overlapeach other, and have a positional relationship in which both are easy tovisually recognize.

The processes described so far are performed in a case where the imagingangle-of-view frame 21 and the enlarged image 22 can be appropriatelydisplayed. However, there are cases where these images cannot bedisplayed or it is better not to display these images.

In the description below, such exceptional processes are explained.

First, a case where the range of the field of view of the imaging device1B cannot be detected in the peripheral field image 20 in step S101 inFIG. 6 is described.

In this case, the image processing device 3 proceeds from step S102 tostep S120, and performs eye proximity sensing determination.

The eye proximity herein refers to a case where the user brings theimaging device 1B immediately before the imaging device 1A (that is, infront of the eyes of the user) as shown in FIG. 2B.

In step S120, the image processing device 3 determines whether or notsuch an eye proximity state is detected.

Various methods may be used as the method for this determination.

For example, in a case where another object exists immediately beforethe lens of the imaging unit 32 using the distance measuring sensor ofthe imaging device 1A, there is a high possibility of an eye proximitystate.

Also, in an eye proximity state, the peripheral field image 20 beingcaptured by the imaging device 1A may include the back surface side ofthe imaging device 1B in a wide range of the screen (or is in anout-of-focus state) as shown in FIG. 11A, for example. Therefore, astate in which an image presumed to be an image of the imaging device 1Bis included in a wide range in the captured image can be determined tobe an eye proximity state by image analysis.

Further, in a case where the imaging device 1B held with a hand of theuser as shown in FIG. 2 performs imaging, it is also possible to acquireinformation about the angular velocity sensor, the acceleration sensor,the posture sensor, and the like of the imaging device 1B, and presumethat the imaging device 1B is at a position or in a posture in an eyeproximity state as shown in FIG. 2B.

Alternatively, a contact sensor may be provided on the front surface ofthe imaging device 1A to detect contact with an object such as theimaging device 1B.

Further, in a case where the imaging devices 1A and 1B include a highlyaccurate location sensor, eye proximity can be detected through thepositional relationship between them.

The image processing device 3 can determine an eye proximity state byone or a combination of a plurality of these methods, for example.

In a case where an eye proximity state is determined, the imageprocessing device 3 proceeds from step S121 to step S122, and performsdisplay control only on the enlarged image 22.

That is, in the display device 2, the peripheral field image 20 and theimaging angle-of-view frame 21 are not displayed, but only the enlargedimage 22 is displayed at the center in the screen as shown in FIG. 11B,for example.

When the peripheral field image 20 becomes as shown in FIG. 11A in aneye proximity state, for example, the periphery is not appropriatelydisplayed, and the image of the imaging field of view of the imagingdevice 1B is not included either. Accordingly, the imaging angle-of-viewframe 21 cannot be displayed either.

Therefore, only the enlarged image 22 as the image being captured by theimaging device 1B is displayed. Particularly, in the case of the postureshown in FIG. 2B, the user feels as if looking into the viewfinder ofthe imaging device 1B, displaying only the image being captured by theimaging device 1B does not cause any strangeness.

On the other hand, in a case where an eye proximity state is notdetermined in step S120, the process being performed by the imageprocessing device 3 proceeds from step S121 to step S123.

This is considered to be a case where the imaging directions of theimaging devices 1A and 1B are greatly different from each other, and theimage of the imaging field of view of the imaging device 1B does notexist in the image (the peripheral field image 20) being captured by theimaging device 1A, as shown in FIG. 2C, for example.

Therefore, the image processing device 3 performs control so that theperipheral field image 20 and the enlarged image 22 are displayed asshown in FIG. 12A, for example.

In this case, as a matter of course, the imaging angle-of-view frame 21is not displayed. This is because there exists no corresponding portion.The enlarged image 22 is displayed at a fixed position in the peripheralfield image 20. In the example in FIG. 12A, the enlarged image 22 issuperimposed and displayed on the right corner portion of the peripheralfield image 20.

Note that, in such a case, the image processing device 3 may performcontrol so that only the peripheral field image 20 is displayed, asshown in FIG. 12 B.

In the display shown in FIG. 12B, when the user is in the posture shownin FIG. 2C, for example, it is easy for the user to recognize a state inwhich the field of view of the imaging device 1B is not included in theperipheral field image 20 substantially corresponding to the user'sfield of view.

Conversely, in the display shown in FIG. 12A, the object being capturedby the imaging device 1B can be recognized in a case where the userchecks the peripheral field of view with the user's face in anydirection.

Since both examples have advantages, in a case where the process moveson to step S123, the user may be allowed to select whether to have theenlarged image 22 displayed in a fixed manner as shown in FIG. 12A, orwhether to have the enlarged image 22 hidden as shown in FIG. 12B.

Next, the process related to the size of the imaging angle-of-view frame21 in step S104 in FIG. 6 is described.

Even in a case where the range of the imaging angle-of-view frame 21 inthe peripheral field image 20 is successfully determined, if the size isdetermined to be “large size” in step S104, the image processing device3 moves on to step S110.

The “large size” of the imaging angle-of-view frame 21 refers to a stateshown in FIG. 13A. That is, it is a case where the range surrounded bythe imaging angle-of-view frame 21 is larger than the enlarged image 22.

Specifically, in step S104, it is only required to determine whether ornot the following is satisfied:

W×H≤Wz×Hz

Note that the size “Wz×Hz” of the enlarged image 22 may be a fixed size.

If W×H≤Wz×Hz is not satisfied, the process moves on to step S105, anddisplay shown in FIG. 10B, FIG. 10C, or FIG. 10D is performed. In thiscase, the enlarged image 22 is larger than the imaging angle-of-viewframe 21, and therefore, it is meaningful to display the enlarged image22. That is, as described above, the user can visually recognize thedetails of the image being captured by the imaging device 1B.

On the other hand, when W×H≤Wz×Hz is satisfied, if processing isperformed in a similar manner, the display shown in FIG. 13A isobtained, and there will be no point in displaying the enlarged image22. This is because the image being captured by the imaging device 1Bcan visually recognized in detail through the image in the range of theimaging angle-of-view frame 21 (that is, part of the image beingcaptured by the imaging device 1A).

Therefore, the image processing device 3 moves on to step S110, andperforms display control on the peripheral field image 20 and theimaging angle-of-view frame 21 while hiding the enlarged image 22. Thatis, the display shown in FIG. 13B is performed in the display device 2.

Note that such a state may correspond to a case where the angle-of-viewdifference between the imaging devices 1A and 1B becomes smaller as theimaging device 1B performs wide-angle imaging, for example.

Also, in step S104, in addition to the determination as to whetherW×H≤Wz×Hz,

a check may be made to determine whether

W×H≤(threshold thS). That is, when the size of the imaging angle-of-viewframe 21 is equal to or larger than a predetermined size, the processmay move on to step S110.

The threshold thS as the predetermined size in this case is notnecessarily the size of the enlarged image 22. For example, even if theimaging angle-of-view frame 21 is slightly smaller than the enlargedimage 22, the significance of displaying the enlarged image 22 might bereduced when the imaging angle-of-view frame 21 is relatively large.Therefore, in a case where the imaging angle-of-view frame 21 has a sizeequal to or larger than a certain size, and the object in the frame canbe visually recognized in sufficient detail, the process moves on tostep S110, and the enlarged image 22 may be hidden.

Next, the process in step S106 in FIG. 6 is described. When the layoutposition of the enlarged image 22 is calculated in accordance with theposition/size of the imaging angle-of-view frame 21 in step S105, theremay be a case where the enlarged image 22 has a portion outside theperipheral field image 20, as shown in FIG. 14A, for example. The sizeof the outside portion varies. A region of less than ½ of the enlargedimage 22 might be located outside as shown in FIG. 14B, or a region of ½or more of the enlarged image 22 might be located outside as shown inFIG. 14B.

Basically, whether to dispose the enlarged image 22 on the right side orthe left side is determined depending on the areas AR1, AR2, and AR3 asdescribed above. However, since the layout position is set after anoffset amount offset_x from the imaging angle-of-view frame 21 issecured, an outside portion as shown in FIG. 14A, 14B, or 14C is likelyto occur when the size of the imaging angle-of-view frame 21 becomeslarger.

The allowable range may be set for such outside portions in variousmanners.

For example, in a case where an outside portion of less than ½ of theregion of the enlarged image 22 is set as the allowable range, theoutside portions shown in FIGS. 14A and 14B are determined to in theallowable range, and the process moves on to step S107. On the otherhand, the outside portion shown in FIG. 14C is regarded unacceptable,and the process moves on to step S110. As described above, in step S110,enlarged image 22 is hidden (see FIG. 13B).

With this arrangement, in a case where the enlarged image 22 has aportion outside the allowable range, the enlarged image 22 is notdisplayed, so that display with poor appearance is not performed.Further, as long as the setting of the areas AR1, AR2, and AR3, and thecorresponding calculation formulas for the enlarged image 22 areappropriate, it can be said that an outside portion appears when thesize of the imaging angle-of-view frame 21 increases to a certainextent. Therefore, there also are circumstances in which it is easy tocheck the image being captured by the imaging device 1B through theimage in the range of the imaging angle-of-view frame 21, withoutforcibly displaying the enlarged image 22.

As the above processes shown in FIG. 6 are performed, the imagingangle-of-view frame 21 and the enlarged image 22 are appropriatelydisplayed in the peripheral field image 20 without any overlap. Also,each of the images, which are the peripheral field image 20, the imagingangle-of-view frame 21, and the enlarged image 22, is displayed inaccordance with the situation.

3. Second Embodiment

An example in which a focus frame is used is now described as a secondembodiment.

For example, as shown in FIG. 15A, a focus frame 23 is displayed inplace of the imaging angle-of-view frame 21 in the peripheral fieldimage 20.

The image processing device 3 only has to perform processes related tothe focus frame 23, instead of the above-described processes related tothe imaging angle-of-view frame 21. For example, as shown in FIG. 15B,the functional configuration of the image processing device 3 includes afocus frame combining unit 11A, instead of the imaging angle-of-viewframe combining unit 11 shown in FIG. 1 .

The processes are substantially similar to those in FIG. 6 , and theprocesses related to the imaging angle-of-view frame 21 are simplyreplaced with the processes related to the focus frame 23.

For example, in step S101, the positional information about the focusframe is acquired from the imaging device 1B, and the range in which theposition is located is detected in the peripheral field image 20. Also,in step S103, the size of the focus frame is determined.

Further, in step S107, display control is performed on the peripheralfield image 20, the focus frame 23, and the enlarged image 22. In stepS110, display control is performed on the peripheral field image 20 andthe focus frame 23.

In this case, the layout position of the enlarged image 22 is calculatedon the basis of the focus frame 23. Therefore, the processes shown inFIG. 16 are performed in the enlarged image layout coordinatecalculation in step S105 in FIG. 6 .

In step S160, the image processing device 3 acquires positionalinformation about the focus frame from the imaging device 1B in theperipheral field image 20. In step S161, the area in which the centralcoordinates of the focus frame 23 are located is then searched for. Forexample, a check is made to determine which one of the areas AR1, AR2,and AR3 the center of the focus frame is located.

After that, steps S152, S153, and S154 are similar to those in FIG. 7 .

In this manner, the layout position of the enlarged image 22 can be seton the basis of the focus frame 23, and both the focus frame 23 and theenlarged image 22 can be displayed with excellent visibility.

Note that the focus frame 23 and the enlarged image 22 are displayedwithin the peripheral field image 20 in the above example. However, in acase where the imaging angle-of-view frame 21 is displayed withoutdisplay of the focus frame 23, the layout position of the enlarged image22 can also be set on the basis of the focus frame 23 in the processesshown in FIG. 16 .

Also, in an example display, the imaging angle-of-view frame 21, thefocus frame 23, and the enlarged image 22 may be displayed within theperipheral field image 20.

Further, in a case where either the imaging angle-of-view frame 21 orthe focus frame 23 is displayed within the peripheral field image 20,the layout position of the enlarged image 22 can also be calculated,using information from which the position of the object can be detected,other than the above information. For example, the information mayinclude the feature points, the specific shape, the color information,and the predicted movement position of the object of the imaging device1B.

4. Third Embodiment

A third embodiment described below is an example in which the right/leftposition of the enlarged image 22 is not frequently switched in a casewhere the area is divided and the right/left position of the enlargedimage 22 is determined as in the first embodiment.

In this case, the processes to be performed are similar to those shownin FIG. 6 . The image processing device 3 performs the processes shownin FIG. 17 as the process of the enlarged image layout coordinatecalculation in step S105 in FIG. 6 .

The image processing device 3 acquires imaging angle-of-view frameinformation in step S150 in FIG. 17 . That is, the respective coordinatevalues of the center and the four corners of the imaging angle-of-viewframe 21 shown in FIG. 9 are acquired on the basis of information aboutthe field of view obtained in step S101 in FIG. 6 .

In step S151 in FIG. 17 , the image processing device 3 searches for thecenter of the imaging angle-of-view frame 21, which is the area in whichthe central coordinates (x0, y0) are located. For example, a search isperformed to determine which one of the areas AR1, AR2, and AR3 shown inFIG. 10A the center is located.

In a case where the central coordinates (x0, y0) are located in the areaAR1, the image processing device 3 moves on to step S173 after stepsS170 and S171, and calculates the layout coordinates of the enlargedimage 22 using the calculation formulas set for the area AR1.

The calculation formulas for the area AR1 are the calculation formulasdescribed as the calculation formulas for the areas AR1 and AR2 in thefirst embodiment.

In a case where the central coordinates (x0, y0) are located in the areaAR3, the image processing device 3 moves on to step S174 after stepsS170 and S171, and calculates the layout coordinates of the enlargedimage 22 using the calculation formulas set for the area AR3. That is,the calculation formulas for the area AR3 are similar to those of thefirst embodiment.

In a case where the central coordinates (x0, y0) are located in thecentral area AR2, the image processing device 3 proceeds from step S170to step S172, and branches the processes depending on which ones of thecalculation formulas for the area AR1 and the calculation formulas forthe area AR3 have been used to calculate the layout coordinates in theprevious frame.

Specifically, in a case where the calculation formulas for the area AR1have been used in the previous frame, the process moves on to step S173,and the layout coordinates of the enlarged image 22 are calculated usingthe calculation formulas set for the area AR1.

In a case where the calculation formulas for the area AR3 have been usedin the previous frame, on the other hand, the process moves on to stepS174, and the layout coordinates of the enlarged image 22 are calculatedusing the calculation formulas set for the area AR3.

That is, in this example, the area AR2 functions as a buffer area havinghysteresis in the right-left position switching.

For example, in a state in which the imaging angle-of-view frame 21 islocated in the area AR1 in a certain frame while the enlarged image 22is displayed on the right side, even if the imaging angle-of-view frame21 gradually moves to the right frame by frame and enters the area AR2,the enlarged image 22 remains displayed on the right side. After that,when the imaging angle-of-view frame 21 further moves to the right andenters area AR3, the enlarged image 22 is displayed on the left side.

Also, in a state in which the imaging angle-of-view frame 21 is locatedin the area AR3 in a certain frame while the enlarged image 22 isdisplayed on the left side, even if the imaging angle-of-view frame 21gradually moves to the left frame by frame and enters the area AR2, theenlarged image 22 remains displayed on the left side. After that, whenthe imaging angle-of-view frame 21 further moves to the left and entersarea AR1, the enlarged image 22 is displayed on the right side.

In this manner, the display position of the enlarged image 22 isprevented from being frequently switched between the right side and leftside on the area boundaries. As a result, any difficulty in viewing isnot caused by frequent right-left switching.

5. Fourth Embodiment

A fourth embodiment is now described with reference to FIG. 18 .

This is an example in which the area is divided into two areas AR1 andAR3 by a segmentation boundary BD0 as shown in FIG. 18A. In thisexample, the segmentation boundary BD0 is set at a position slightly tothe right of the center in the horizontal direction.

In a case where the central coordinates (x0, y0) of the imagingangle-of-view frame 21 are located in the area AR1, the enlarged image22 is disposed on the right side of the imaging angle-of-view frame 21as shown in FIGS. 18B and 18C.

In a case where the central coordinates (x0, y0) of the imagingangle-of-view frame 21 are located in the area AR3, on the other hand,the enlarged image 22 is disposed on the left side of the imagingangle-of-view frame 21 as shown in FIG. 18D.

That is, this is a concept in which the areas AR1 and AR2 of the firstembodiment are combined into the area AR1.

As the area is divided into at least two areas in this manner, theposition of the enlarged image 22 can be appropriately switched betweenthe right side and the left side.

Note that, in a case where hysteresis is applied to the right-leftswitching as in the third embodiment, a line HT may be set as shown inFIG. 18E, for example, and the right side of the line HT in the area AR1is regarded as an area similar to the area AR2 in the third embodiment.

6. Fifth Embodiment

A fifth embodiment described below is an example in which the inside ofthe peripheral field image 20 is divided into nine areas: three areas inthe horizontal direction and three areas in the vertical direction.

For example, as shown in FIG. 19A, areas AR1 to AR9 are set by verticalboundary lines BD1 and BD2 and horizontal boundary lines BD3 and BD4.

In this case, the image processing device may perform the processesshown in FIG. 6 . However, in the process of the enlarged image layoutcoordinate calculation in step S105, the layout position of the enlargedimage 22 is calculated using calculation formulas corresponding to thearea in which the central coordinates (x0, y0) of the imagingangle-of-view frame 21 are located among the nine areas AR1 to AR9.

In particular, in this case, not only the right/left position of theenlarged image 22 but also the vertical position thereof is switched.

Specifically, in a case where the central coordinates (x0, y0) of theimaging angle-of-view frame 21 belong to one of the areas AR4, AR5, andAR6, which are the central areas in the vertical direction, the enlargedimage 22 is disposed with a horizontal offset amount offset_x on theright side or the left side of the imaging angle-of-view frame 21. FIG.19B shows an example in which the enlarged image 22 is disposed on theright side.

Also, in a case where the central coordinates (x0, y0) of the imagingangle-of-view frame 21 belong to one of the areas AR7, AR8, and AR9,which are the lower areas in the vertical direction, the enlarged image22 is disposed with the horizontal offset amount offset_x and a verticaloffset amount offset_y on the upper right side or the upper left side ofthe imaging angle-of-view frame 21. FIG. 19C shows an example in whichthe enlarged image 22 is disposed on the upper right side.

Further, in a case where the central coordinates (x0, y0) of the imagingangle-of-view frame 21 belong to one of the areas AR1, AR2, and AR3,which are the upper areas in the vertical direction, the enlarged image22 is disposed with the horizontal offset amount offset_x and thevertical offset amount offset_y on the lower right side or the lowerleft side of the imaging angle-of-view frame 21. FIG. 19D shows anexample in which the enlarged image 22 is disposed on the lower rightside.

FIG. 20 shows an example process in step S105 in FIG. 6 for such alayout.

The image processing device 3 acquires imaging angle-of-view frameinformation in step S150 in FIG. 20 . That is, the respective coordinatevalues of the center and the four corners of the imaging angle-of-viewframe 21 shown in FIG. 9 are acquired on the basis of information aboutthe field of view obtained in step S101 in FIG. 6 .

In step S151 in FIG. 20 , the image processing device 3 searches for thecenter of the imaging angle-of-view frame 21, which is the area in whichthe central coordinates (x0, y0) are located. For example, a search isperformed to determine which one of the areas AR1 to AR9 shown in FIG.19A the center is located.

In a case where the central coordinates (x0, y0) are located in one ofthe areas AR1, AR2, and AR3, the image processing device 3 proceeds fromstep S201 to step S210.

In a case where the central coordinates (x0, y0) are located in the areaAR1, the image processing device 3 then moves on to step S213 after stepS211, and calculates the layout coordinates of the enlarged image 22using the calculation formulas set for the area AR1 described later.

In a case where the central coordinates (x0, y0) are located in the areaAR3, on the other hand, the image processing device 3 moves on to stepS214 after step S211, and calculates the layout coordinates of theenlarged image 22 using the calculation formulas set for the area AR3described later.

In a case where the central coordinates (x0, y0) are located in thecentral area AR2, the image processing device 3 proceeds from step S210to step S212, and branches the processes depending on which ones of thecalculation formulas for the left areas and the calculation formulas forthe right areas have been used to calculate the layout coordinates inthe previous frame.

In this case, the calculation formulas for the left areas refer to thecalculation formulas for the area AR1, the calculation formulas for thearea AR4, and the calculation formulas for the area AR7.

Also, the calculation formulas for the right areas refer to thecalculation formulas for the area AR3, the calculation formulas for thearea AR6, and the calculation formulas for the area AR9.

In a case where the calculation formulas for the left areas have beenused in the previous frame, the process moves on to step S213, and thelayout coordinates of the enlarged image 22 are calculated using thecalculation formulas set for the area AR1.

In a case where the calculation formulas for the right areas have beenused in the previous frame, on the other hand, the process moves on tostep S214, and the layout coordinates of the enlarged image 22 arecalculated using the calculation formulas set for the area AR3.

In a case where the central coordinates (x0, y0) belong to one of theareas AR4, AR5, and AR6, the image processing device 3 moves on to stepS220 after steps S201 and S202.

In a case where the central coordinates (x0, y0) are located in the areaAR4, the image processing device 3 then moves on to step S223 after stepS221, and calculates the layout coordinates of the enlarged image 22using the calculation formulas set for the area AR4 described later.

In a case where the central coordinates (x0, y0) are located in the areaAR6, on the other hand, the image processing device 3 moves on to stepS224 after step S221, and calculates the layout coordinates of theenlarged image 22 using the calculation formulas set for the area AR6described later.

In a case where the central coordinates (x0, y0) are located in thecentral area AR5, the image processing device 3 proceeds from step S220to step S222, and branches the processes depending on which ones of thecalculation formulas for the left areas and the calculation formulas forthe right areas have been used to calculate the layout coordinates inthe previous frame.

In a case where the calculation formulas for the left areas have beenused in the previous frame, the process moves on to step S223, and thelayout coordinates of the enlarged image 22 are calculated using thecalculation formulas set for the area AR4.

In a case where the calculation formulas for the right areas have beenused in the previous frame, on the other hand, the process moves on tostep S224, and the layout coordinates of the enlarged image 22 arecalculated using the calculation formulas set for the area AR6.

In a case where the central coordinates (x0, y0) belong to one of theareas AR7, AR8, and AR9, the image processing device 3 moves on to stepS230 after steps S201 and S202.

In a case where the central coordinates (x0, y0) are located in the areaAR7, the image processing device 3 then moves on to step S233 after stepS231, and calculates the layout coordinates of the enlarged image 22using the calculation formulas set for the area AR7 described later.

In a case where the central coordinates (x0, y0) are located in the areaAR9, on the other hand, the image processing device 3 moves on to stepS234 after step S231, and calculates the layout coordinates of theenlarged image 22 using the calculation formulas set for the area AR9described later.

In a case where the central coordinates (x0, y0) are located in thecentral area AR8, the image processing device 3 proceeds from step S230to step S232, and branches the processes depending on which ones of thecalculation formulas for the left areas and the calculation formulas forthe right areas have been used to calculate the layout coordinates inthe previous frame.

In a case where the calculation formulas for the left areas have beenused in the previous frame, the process moves on to step S233, and thelayout coordinates of the enlarged image 22 are calculated using thecalculation formulas set for the area AR7.

In a case where the calculation formulas for the right areas have beenused in the previous frame, on the other hand, the process moves on tostep S234, and the layout coordinates of the enlarged image 22 arecalculated using the calculation formulas set for the area AR9.

The calculation formulas for calculating the coordinates of the fourcorners of the enlarged image 22 depending on the respective areas areshown below.

First, as an example, example calculations based on the end sides of theimaging angle-of-view frame 21 are described.

Note that the definitions of the respective coordinates are as shown inFIG. 9 .

Further, “xr” represents the x-coordinate value of the right end side ofthe imaging angle-of-view frame 21, “xl” represents the x-coordinatevalue of the left end side of the imaging angle-of-view frame 21, “yu”represents the y-coordinate value of the upper end side of the imagingangle-of-view frame 21, and “yd” is the y-coordinate value of the lowerend side of the imaging angle-of-view frame 21 (see FIGS. 19B, 19C, and19D).

Calculation Formulas for the Area AR1

(Xul,Yul)=(xr+offset_x,yd−offset_y)

(Xur,Yur)=(xr+offset_x+Wz,yd−offset_y)

(Xdl,Ydl)=(xr+offset_x,yd−offset_y−Hz)

(Xdr,Ydr)=(xr+offset_x+Wz,yd−offset_y−Hz)

Calculation Formulas for the Area AR3

(Xul,Yul)=(xl−offset_x−Wz,yd−offset_y)

(Xur,Yur)=(xl−offset_x,yd−offset_y)

(Xdl,Ydl)=(xl−offset_x−Wz,yd−offset_y−Hz)

(Xdr,Ydr)=(xl−offset_x,yd−offset_y−Hz)

Calculation Formulas for the Area AR4

(Xul,Yul)=(xr+offset_x,y0+Hz/2)

(Xur,Yur)=(xr+offset_x+Wz,y0+Hz/2)

(Xdl,Ydl)=(xr+offset_x,y0−Hz/2)

(Xdr,Ydr)=(xr+offset_x+Wz,y0−Hz/2)

Calculation Formulas for the Area AR6

(Xul,Yul)=(xl−offset_x−Wz,y0+Hz/2)

(Xur,Yur)=(xl−offset_x,y0+Hz/2)

(Xdl,Ydl)=(xl−offset_x−Wz,y0−Hz/2)

(Xdr,Ydr)=(xl−offset_x,y0−Hz/2)

Calculation Formulas for the Area AR7

(Xul,Yul)=(xr+offset_x,yu+offset_y+Hz)

(Xur,Yur)=(xr+offset_x+Wz,yu+offset_y+Hz)

(Xdl,Ydl)=(xr+offset_x,yu+offset_y)

(Xdr,Ydr)=(xr+offset_x+Wz,yu+offset_y)

Calculation Formulas for the Area AR9

(Xul,Yul)=(xl−offset_x−Wz,yu+offset_y+Hz)

(Xur,Yur)=(xl−offset_x,yu+offset_y+Hz)

(Xdl,Ydl)=(xl−offset_x−Wz,yu+offset_y)

(Xdr,Ydr)=(xl−offset_x,yu+offset_y)

Instead of the above, example calculations based on the centralcoordinates (x0, y0) of the imaging angle-of-view frame 21 can be usedin the layout coordinate calculations for disposing the enlarged image22.

For the respective areas, the central coordinates (X0, Y0) of theenlarged image 22 are calculated from the central coordinates (x0, y0)of the imaging angle-of-view frame 21 as shown below.

Calculation Formulas for the Area AR1

(X0,Y0)=(x0+W/2+offset_x+Wz/2,y0−H/2−offset_y−Hz/2)

Calculation Formulas for the Area AR3

(X0,Y0)=(x0−W/2−offset_x−Wz/2,y0−H/2−offset_y−Hz/2)

Calculation Formulas for the Area AR4

(X0,Y0)=(x0+W/2+offset_x+Wz/2,y0)

Calculation Formulas for the Area AR6

(X0,Y0)=(x0−W/2−offset_x−Wz/2,y0)

Calculation Formulas for the Area AR7

(X0,Y0)=(x0+W/2+offset_x+Wz/2,y0+H/2+offset_y+Hz/2)

Calculation Formulas for the Area AR9

(X0,Y0)=(x0−W/2−offset_x−Wz/2,y0+H/2+offset_y+Hz/2)

Further, in each case, the coordinates of the four corners of theenlarged image 22 are calculated according to the equations shown below.

(Xul,Yul)=(X0−Wz/2,Y0+Hz/2)

(Xur,Yur)=(X0+Wz/2,Y0+Hz/2)

(Xdl,Ydl)=(X0−Wz/2,Y0−Hz/2)

(Xdr,Ydr)=(X0+Wz/2,Y0−Hz/2)

As the processes shown in FIG. 20 are performed using the above examplecalculation formulas, the enlarged image 22 is disposed as follows.

-   -   When the central coordinates (x0, y0) of the imaging        angle-of-view frame 21 are located in the area AR1, the enlarged        image 22 is disposed on the lower right side, with the        horizontal offset amount offset_x from the right end side and        the vertical offset amount offset_y from the lower end side of        the imaging angle-of-view frame 21 being maintained.    -   When the central coordinates (x0, y0) of the imaging        angle-of-view frame 21 are located in the area AR3, the enlarged        image 22 is disposed on the lower left side, with the horizontal        offset amount offset_x from the left end side and the vertical        offset amount offset_y from the lower end side of the imaging        angle-of-view frame 21 being maintained.    -   When the central coordinates (x0, y0) of imaging angle-of-view        frame 21 are located in the area AR2, the enlarged image 22 is        disposed either on the lower right side or the lower left side        as above.    -   When the central coordinates (x0, y0) of the imaging        angle-of-view frame 21 are located in the area AR4, the enlarged        image 22 is disposed on the right side, with the horizontal        offset amount offset_x from the right end side of the imaging        angle-of-view frame 21 being maintained.    -   When the central coordinates (x0, y0) of the imaging        angle-of-view frame 21 are located in the area AR6, the enlarged        image 22 is disposed on the left side, with the horizontal        offset amount offset_x from the left end side of the imaging        angle-of-view frame 21 being maintained.    -   When the central coordinates (x0, y0) of imaging angle-of-view        frame 21 are located in the area AR5, the enlarged image 22 is        disposed either on the right side or the left side as above.    -   When the central coordinates (x0, y0) of the imaging        angle-of-view frame 21 are located in the area AR7, the enlarged        image 22 is disposed on the upper right side, with the        horizontal offset amount offset_x from the right end side and        the vertical offset amount offset_y from the upper end side of        the imaging angle-of-view frame 21 being maintained.    -   When the central coordinates (x0, y0) of the imaging        angle-of-view frame 21 are located in the area AR9, the enlarged        image 22 is disposed on the upper left side, with the horizontal        offset amount offset_x from the left end side and the vertical        offset amount offset_y from the upper end side of the imaging        angle-of-view frame 21 being maintained.    -   When the central coordinates (x0, y0) of imaging angle-of-view        frame 21 are located in the area AR8, the enlarged image 22 is        disposed either on the upper right side or the upper left side        as above.

As the layout position of the enlarged image 22 is set as describedabove, for example, the imaging angle-of-view frame 21 and the enlargedimage 22 are appropriately displayed within the peripheral field image20 without any overlap. Particularly, the enlarged image 22 is disposedon the upper, lower, right, or left side of the imaging angle-of-viewframe 21, depending on the areas, and is disposed at a predetermineddistance from the imaging angle-of-view frame 21, the predetermineddistance being expressed as the offset amounts offset_x and offset_y.Thus, a display state in which a high visibility is maintained for theuser can be achieved.

Further, as for the directions of the right and left positions, thecentral areas AR2, AR5, and AR8 aligned in a horizontal directionfunction as buffer areas applying hysteresis to the right-left positionswitching of the enlarged image 22. Thus, right-left switching does notoccur frequently, which is suitable for enhancing visibility.

Note that hysteresis may be applied to switching of positions in thevertical direction.

Note that, in a case where the processes in steps S103, S104, S106,S110, and S120 to S123 in FIG. 6 are also performed, the enlarged image22 is hidden, the peripheral field image 20 is hidden, or only theenlarged image is displayed, as an exceptional process in the fifthembodiment. With this arrangement, display suitable for the visual fieldstates and the imaging directions of the imaging devices 1A and 1B isperformed.

FIG. 21 shows a modification of the area segmentation. In this example,the area AR5 has an elliptical shape, and the area segmentation may beset using such a curved line. Various other examples of areasegmentation are of course conceivable.

7. Sixth Embodiment

A sixth embodiment described below is an example that does not involvethe area segmentation setting described in the first to fifthembodiments.

However, the enlarged image 22 is also located at a certain distancefrom the imaging angle-of-view frame 21, and the enlarged image 22 alsomoves horizontally as the imaging angle-of-view frame 21 moves, as inthe above embodiments.

However, whether the enlarged image 22 is disposed on the right side orthe left side of the imaging angle-of-view frame 21 is determined in amanner similar that of the previous frame, and right-left switching isperformed under a specific condition.

FIG. 22A shows a state in which the enlarged image 22 is disposed on theright side of the imaging angle-of-view frame 21.

FIG. 22B shows a state in which the enlarged image 22 is located at aposition shifted to the right when the imaging angle-of-view frame 21shifts to the right.

In either case, the imaging angle-of-view frame 21 and the enlargedimage 22 have positional relationship involving a horizontal offsetamount offset_x as shown in FIG. 23A.

If the layout position of the enlarged image 22 is determined to beinappropriate at a certain point of time, the layout position of theenlarged image 22 is horizontally inverted, and the enlarged image 22 isdisposed on the left side of the imaging angle-of-view frame 21 as shownin FIG. 22C, for example.

In this case, the imaging angle-of-view frame 21 and the enlarged image22 also have positional relationship involving the horizontal offsetamount offset_x as shown in FIG. 23B.

As described above, the enlarged image 22 follows the imagingangle-of-view frame 21 while maintaining the horizontal offset amountoffset_x, and the right/left position thereof is changed in accordancewith the position of the imaging angle-of-view frame 21 in a mannersubstantially similar to that in the first embodiment. However, thechange in the right/left position is not based on the area segmentationsetting in this case.

FIG. 24 shows example processes to be performed by the image processingdevice 3 according to the sixth embodiment.

Note that, in FIG. 24 , processes similar to those in FIG. 6 are denotedby the same step numbers as those in FIG. 6 , and explanation of themwill not be repeated below. The differences from FIG. 6 are that, instep S105A, the process of the enlarged image layout coordinatecalculation is performed in a manner similar to that in the previousframe, and, in a case where the enlarged image layout coordinates aredetermined not to be appropriate, the enlarged image layout coordinatesare recalculated in step S131.

In step S105A, the enlarged image layout coordinate calculation isperformed. In this case, if the enlarged image 22 is disposed on theright side of the imaging angle-of-view frame 21 in the previous frame,the calculation is also performed for a right-side position in thecurrent frame. That is, the central coordinates (X0, Y0) of the enlargedimage 22 are calculated according to the following equation:

(X0,Y0)=(x0+W/2+offset_x+Wz/2,y0)

The coordinate values of the four corners of the enlarged image 22 arethen calculated as follows.

(Xul,Yul)=(X0−Wz/2,Y0+Hz/2)

(Xur,Yur)=(X0+Wz/2,Y0+Hz/2)

(Xdl,Ydl)=(X0−Wz/2,Y0−Hz/2)

(Xdr,Ydr)=(X0+Wz/2,Y0−Hz/2)

Further, in a case where the enlarged image 22 is located on the leftside of the imaging angle-of-view frame 21 in the previous frame at thetime of step S105A, the calculation is also performed for a left-sideposition in the current frame. That is, the central coordinates (X0, Y0)of the enlarged image 22 are calculated according to the followingequation:

(X0,Y0)=(x0−W/2−offset_x−Wz/2,y0)

The coordinate values (Xul, Yul), (Xur, Yur), (Xdl, Ydl), and (Xdr, Ydr)of the four corners of the enlarged image 22 are then calculated in amanner similar to the above.

As a result, the displays as shown in FIGS. 22A, 22B, and 22C, forexample, are obtained through the display control in step S107.

By the calculation in step S105A, however, the layout position of theenlarged image 22 becomes inappropriate in some cases. That is, theenlarged image 22 has a portion outside the peripheral field image 20 insome cases.

In that case, the image processing device 3 proceeds from step S130 tostep S131, and performs a layout coordinate recalculation process asshown in FIG. 25 .

Note that the sixth embodiment is an example in which only an outsideportion in a horizontal direction (rightward/leftward direction) istaken into consideration. An example in which only an outside portion ina vertical direction (upward/downward direction) is taken intoconsideration at the time of the layout coordinate recalculation will bedescribed later as a seventh embodiment.

In a case where the layout position of the enlarged image 22 calculatedin step S105A has a portion outside the peripheral field image 20, andthe process moves on to step S131, the process first proceeds from stepS301 to step S302 in FIG. 25 .

In this case, the layout position of the enlarged image 22 ishorizontally inverted and is recalculated. For example, in a case wherean outside portion has appeared at the time of the calculation for aright-side position as above, recalculation is performed for a left-sideposition as above.

The process then returns to step S130 in FIG. 24 , as indicated by “c1”.If the enlarged image 22 no longer has a portion outside the peripheralfield image 20 as a result of the horizontal inversion performed in sucha manner, the enlarged image layout position is determined to be “OK”,and the process moves on to step S107.

FIG. 26A shows a state in which the layout position of the enlargedimage 22 has a portion outside the peripheral field image 20 ascalculated in step S105A, for example. FIG. 26B shows a state in whichthe layout position of the enlarged image 22 is horizontally inverted tobe located within the peripheral field image 20.

However, even if the horizontal inversion is performed, the layoutposition of the enlarged image 22 might still be inappropriate.

In that case, the image processing device 3 again proceeds from stepS130 to step S131, and performs the processes shown in FIG. 25 . In thiscase, the horizontal inversion has already been performed, andtherefore, the image processing device 3 proceeds from step S301 to stepS303. Since any shift has not been performed yet, the layout position isthen recalculated in step S304 so that the layout position of theenlarged image 22 shifts.

For example, the value of the horizontal offset amount offset_x isreduced, and recalculation is then performed by the calculation formulasfor a right-side position or the calculation formulas for a left-sideposition as described above.

That is, the value of the horizontal offset amount offset_x is changedas follows:

offset_x=offset_x−Δx,

and recalculation is then performed.

The value of Δx may be a fixed value, or may represent the amount of theoutside portion in the x-axis direction.

The process then returns to step S130 in FIG. 24 , as indicated by “c1”.If the enlarged image 22 no longer has a portion outside the peripheralfield image 20 as a result of the horizontal shift performed in such amanner, the enlarged image layout position is determined to be “OK”, andthe process moves on to step S107.

FIG. 27A shows that, in a case where the layout position of the enlargedimage 22 has an outside portion on the left side of the peripheral fieldimage 20 even after horizontal inversion occurs through recalculation,for example, a horizontal shift to the right is performed so that thelayout position of the enlarged image 22 is changed to a position nothaving any outside portion.

FIG. 27B shows that, in a case where the layout position of the enlargedimage 22 has an outside portion on the right side of the peripheralfield image 20 even after horizontal inversion occurs throughrecalculation, for example, a horizontal shift to the left is performedso that the layout position of the enlarged image 22 is changed to aposition not having any outside portion.

However, even if the horizontal inversion and the shift are performed asdescribed above, the layout position of the enlarged image 22 mightstill be inappropriate. In that case, the process moves on to step S131for the third time, but this time, the process proceeds from step S303in FIG. 25 to step S110 in FIG. 24 as indicated by “c2”. In this case,display control is performed on the peripheral field image 20 and theimaging angle-of-view frame 21 while the enlarged image 22 is hidden.

As described above, in the sixth embodiment, when the enlarged image 22to follow the imaging angle-of-view frame 21 has a portion outside theperipheral field image 20, horizontal inversion is performed. In a casewhere there still is a portion outside the peripheral field image, ashift is performed to counter the problem.

Note that, when the enlarged image 22 has even a small portion outsidethe peripheral field image 20, the recalculation in step S131 may beperformed. However, an outside portion of a certain size may be set asan allowable range in step S130.

8. Seventh Embodiment

A seventh embodiment is an example in which, in a case where the imagingangle-of-view frame 21 moves in a vertical direction, the layoutposition of the enlarged image 22 moves in the opposite direction, inaddition to the sixth embodiment.

For example, in a case where the central coordinates (x0, y0) of theimaging angle-of-view frame 21=(x0, 0) in the coordinate system shown inFIG. 8 , the central coordinates (X0, Y0) of the enlarged image 22=(X0,0), and the layout position of the enlarged image 22 is set exactly in alateral direction of the imaging angle-of-view frame 21 as shown inFIGS. 22A, 22B, and 22C.

In a case where the vertical position of the imaging angle-of-view frame21 shifts upward from the y-coordinate value=0, on the other hand, thelayout position of the enlarged image 22 shifts downward by theequivalent amount as shown in FIG. 28A. However, the horizontal offsetamount offset_x is maintained, as shown in FIG. 29A.

Further, in a case where the vertical position of the imagingangle-of-view frame 21 shifts downward from the y-coordinate value=0,the layout position of the enlarged image 22 shifts upward by theequivalent amount as shown in FIG. 28B. In this case, the horizontaloffset amount offset_x is also maintained, as shown in FIG. 29B.

In such cases, the processes to be performed are similar to those inFIG. 24 . However, in the process of the enlarged image layoutcoordinate calculation in step S105A, the calculations shown below areperformed, for example.

In a case where the enlarged image 22 is located on the right side ofthe imaging angle-of-view frame 21 in the previous frame at the time ofthe enlarged image layout coordinate calculation in step S105A, thecalculation is also performed for a right-side position in the currentframe. That is, the central coordinates (X0, Y0) of the enlarged image22 are calculated according to the following equation:

(X0,Y0)=(x0+W/2+offset_x+Wz/2,0−y0)

That is, the calculation of the Y-coordinate value Y0 at the centralcoordinates (X0, Y0) differs from that of the sixth embodiment.

The coordinate values of the four corners of the enlarged image 22 arethen calculated as follows.

(Xul,Yul)=(X0−Wz/2,Y0+Hz/2)

(Xur,Yur)=(X0+Wz/2,Y0+Hz/2)

(Xdl,Ydl)=(X0−Wz/2,Y0−Hz/2)

(Xdr,Ydr)=(X0+Wz/2,Y0−Hz/2)

Further, in a case where the enlarged image 22 is located on the leftside of the imaging angle-of-view frame 21 in the previous frame at thetime of step S105A, the calculation is also performed for a left-sideposition in the current frame. That is, the central coordinates (X0, Y0)of the enlarged image 22 are calculated according to the followingequation:

(X0,Y0)=(x0−W/2−offset_x−Wz/2,0−y0)

The coordinate values (Xul, Yul), (Xur, Yur), (Xdl, Ydl), and (Xdr, Ydr)of the four corners of the enlarged image 22 are then calculated in amanner similar to the above.

With this arrangement, the layout position of the enlarged image 22 isset so as to follow in a horizontal direction or inversely follow in avertical direction, as shown in FIG. 28 .

In a case where the enlarged image 22 has a portion outside theperipheral field image 20, the image processing device 3 performs thelayout coordinate recalculation process in step S131 as shown in FIG. 30, instead of FIG. 25 .

In a case where the layout coordinate recalculation process is firstperformed in the current frame, the image processing device 3 carriesout step S321 after step S320, and checks whether or not there is anoutside portion in a horizontal direction at the layout position of theenlarged image 22 previously calculated in step S105A.

There is a possibility that an outside portion may occur in a horizontaldirection or a vertical direction.

In a case where an outside portion appears only in a vertical direction,the image processing device 3 moves on to step S324, and performs layoutposition recalculation so that the layout position of the enlarged image22 shifts in a vertical direction.

For example, in a case where there is an outside portion on the upperside as shown in FIG. 31A, the following calculation is performed:

(X0,Y0)=(x0−W/2−offset_x−Wz/2,0−y0−Δy)

On the other hand, in a case where there is an outside portion on thelower side as shown in FIG. 31B, the following calculation is performed:

(X0,Y0)=(x0−W/2−offset_x−Wz/2,0−y0+Ay)

The value of Ay may be a fixed value, or may represent the amount of theoutside portion in the y-axis direction shown in FIGS. 31A and 31B.

The process then returns to step S130 in FIG. 24 , as indicated by “c1”.

In a case where there is an outside portion in a horizontal direction,on the other hand, the image processing device 3 proceeds from step S321to step S325 in FIG. 30 , and checks whether or not there is an outsideportion in a vertical direction at the layout position of the enlargedimage 22 previously calculated in step S105A.

If there is not an outside portion in a vertical direction, the imageprocessing device 3 moves on to step S326, and horizontally inverts thelayout position of the enlarged image 22 and recalculates the layoutposition. For example, in a case where an outside portion has appearedat the time of the calculation for a right-side position as above,recalculation is performed for a left-side position as above.

The process then returns to step S130 in FIG. 24 , as indicated by “c1”.

In a case where there is an outside portion in a vertical direction instep S325 in FIG. 30 , which is a case where an outside portion appearsin both a horizontal direction and a vertical direction, the imageprocessing device 3 moves on to step S327, and performs recalculation sothat the layout position of the enlarged image 22 is horizontallyinverted and shifts in a vertical direction.

The process then returns to step S130 in FIG. 24 , as indicated by “c1”.

If the enlarged image 22 no longer has a portion outside the peripheralfield image 20 as a result of the horizontal inversion and/or thevertical shift performed as above, the enlarged image layout position isdetermined to be “OK”, and the process moves on to step S107 in FIG. 24.

However, even if the horizontal inversion and/or the vertical shift isperformed, the layout position of the enlarged image 22 might still beinappropriate.

In that case, the image processing device 3 again proceeds from stepS130 to step S131 in FIG. 24 , and performs the processes shown in FIG.30 .

In this case, the horizontal inversion has already been performed, andtherefore, the image processing device 3 proceeds from step S320 to stepS323. Since any horizontal shift has not been performed yet, the layoutposition is then recalculated in step S329 so that the layout positionof the enlarged image 22 shifts in a horizontal direction.

For example, after the value of the horizontal offset amount offset_x isreduced (offset_x=offset_x-Ox, for example), recalculation is performedaccording to the above equation for a right-side position or the aboveequation for a left-side position.

The process then returns to step S130 in FIG. 24 , as indicated by “c1”.If the enlarged image 22 no longer has a portion outside the peripheralfield image 20 as a result of the horizontal shift performed in such amanner, the enlarged image layout position is determined to be “OK”, andthe process moves on to step S107.

However, even if the horizontal inversion, the vertical shift, and thehorizontal shift are performed as described above, the layout positionof the enlarged image 22 might still be inappropriate. In that case, theprocess moves on to step S131 for the third time, but this time, theprocess proceeds from step S323 in FIG. 30 to step S110 in FIG. 24 asindicated by “c2”. In this case, display control is performed on theperipheral field image 20 and the imaging angle-of-view frame 21 whilethe enlarged image 22 is hidden.

As described above, in the seventh embodiment, the layout position ofthe enlarged image 22 follows the imaging angle-of-view frame 21 in ahorizontal direction or inversely follows the imaging angle-of-viewframe 21 in a vertical direction. In a case where the enlarged image 22has a portion outside the peripheral field image 20, the enlarged image22 shifts in a vertical direction, inverts in a horizontal direction, orshifts in a horizontal direction.

Note that the layout coordinate recalculation process taking intoconsideration an outside portion in vertical and horizontal directionsas shown in FIG. 30 can also be adopted in the case of the sixthembodiment.

9. Display of an Imaging Angle-of-View Frame

In each of the above embodiments, the imaging angle-of-view frame 21 hasbeen indicated by a rectangle, but is not necessarily a rectangle. Forexample, FIG. 32A shows a non-rectangular imaging angle-of-view frame21.

Particularly, in a case where the imaging directions of the imagingdevice 1A and the imaging device 1B are different, the imagingangle-of-view frame may be shaped in accordance with the difference inthe field-of-view direction, so that the imaging direction of theimaging device 1B can be stereoscopically viewed.

For example, as shown in FIG. 32B, when a viewpoint E is set as theposition of the imaging device 1B, the imaging angle-of-view frame 21may have a shape obtained by projecting the angle of view of the imagingdevice 1B from the viewpoint E.

Note that, even in a case where the imaging angle-of-view frame 21 isnot rectangular as described above, if the imaging angle-of-view frame21 is regarded as a pseudo rectangle as in FIG. 32B, and the size H in avertical direction, the size W in a horizontal direction, and thecentral coordinates (x0, y0) are defined, the layout position of theenlarged image 22 can be calculated by the method of each of the aboveembodiments.

10. Summary and Modifications

In the above embodiments, the effects described below are achieved.

The image processing device 3 according to the first to seventhembodiments includes the image combining unit 12 that performs acombining process of disposing the enlarged image 22 (second image) inthe peripheral field image 20 (first image) captured by the imagingdevice 1A, the enlarged image 22 being an image of the imaging field ofview of the imaging device 1B capable of imaging a partial field of viewin the imaging field of view of the imaging device 1A.

As the enlarged image 22, which is an image of a relatively narrowimaging field of view captured by the imaging device 1B, is displayed onthe peripheral field image 20 of a relatively wide field of viewcaptured by the imaging device 1A, it is possible to check the imagingtarget of the imaging device 1B, with part of the image being enlargedin the peripheral field of view. Thus, the user can easily follow theobject with the imaging device 1B while checking the peripheral field ofview.

For example, in a case where imaging is performed by following part ofan object such as a person or an animal in a wide scene, or in a case ofan object moving very fast in a motor sport or the like, it is difficultto follow the object only by checking a zoomed-in image of the object.As the enlarged image 22 indicating the imaging field of view of theimaging device 1B is displayed in the peripheral field image 20 capturedby the imaging device 1A as in the embodiments, the user can follow thetarget object by zooming while checking a wide field of view. This issuitable in the imaging situation as described above.

Meanwhile, the object that is the target of the imaging device 1B isdisplayed as the enlarged image 22 more finely than other objects in theperiphery. Accordingly, this is also suitable for checking the state ofthe target object.

Also, as the enlarged image 22 is displayed in the peripheral fieldimage 20 having a wider angle, the user who wishes to check thesurrounding situation and the target object at the same time moves lessfrequently his/her line of sight relative to the display, and thus, caneasily view the display.

Further, the peripheral field image 20 is not displayed in the enlargedimage 22, but the enlarged image 22 is displayed in the peripheral fieldimage 20. This is suitable for checking the surrounding situation,without an unnecessary reduction in size of the peripheral field image20.

Note that the enlarged image 22 is combined with the image data of theperipheral image captured by the imaging device 1A using the image datacaptured by the imaging device 1B. However, the image data captured bythe imaging device 1B is not necessarily used. For example, the imagedata in the range of the imaging angle-of-view frame 21 in the imagedata of the peripheral image may be extracted, an interpolation processmay be performed to generate enlarged image data, and a combiningprocess may be performed so that the enlarged image data is superimposedand displayed on the image data of the peripheral image. That is, theenlarged image 22 (second image) may be an image of the imaging field ofview of the imaging device 1B, or is only required to be an imageindicating the field-of-view range being imaged by the imaging device1B.

In the first, third, fourth, fifth, sixth, and seventh embodiments, theimage processing device 3 includes: the range detection unit 10 thatdetects the range of the imaging field of view of the imaging device 1Bin the peripheral field image 20; and the imaging angle-of-view framecombining unit 11 that combines the peripheral field image 20 with theimaging angle-of-view frame 21 indicating the range of the imaging fieldof view detected by the range detection unit 10.

As the imaging angle-of-view frame 21 is displayed in the peripheralfield image 20, it is possible to check which range the imaging device1B is imaging in the peripheral field image 20, and the user can moreeasily follow the target object.

Further, as the imaging angle-of-view frame 21 and the enlarged image 22are displayed, it is possible to check the image of the target object(the detailed situation of the target object) being captured by theimaging device 1B, while checking the position of the target object inthe peripheral field image 20.

Particularly, as the imaging angle-of-view frame 21 and the enlargedimage 22 are displayed in the peripheral field image 20, movement of theuser's line of sight becomes smaller, and it becomes easier to check theperiphery and the target object at the same time.

In the first, third, fourth, fifth, sixth, and seventh embodiments, theimage combining unit 12 sets the layout position of the enlarged image22 in the peripheral field image 20 in accordance with the position ofthe imaging angle-of-view frame 21.

As the imaging angle-of-view frame 21 and the enlarged image 22 aredisplayed in the peripheral field image 20, the user can check theimaging situation of the target object while checking which range theimaging device 1B is imaging in the peripheral field image 20. However,it is important that these images are appropriately disposed anddisplayed.

Specifically, the position of the imaging angle-of-view frame 21 in theperipheral field image 20 changes depending on the respective viewingdirections of the imaging device 1A and the imaging device 1B, and thesize of the imaging angle-of-view frame 21 changes with the zoomingstate (the angle of view) of the imaging device 1B. Therefore, if thedisplay position of the enlarged image 22 is fixed, the imagingangle-of-view frame 21 and the enlarged image 22 might overlap eachother.

To counter this, the position of the enlarged image 22 in the peripheralfield image 20 is made variable with the position of the imagingangle-of-view frame 21, as described above in each of the embodiments.As a result, the imaging angle-of-view frame 21 and the enlarged image22 are disposed and displayed in an easily viewable state withoutoverlapping each other, regardless of changes in the viewing directionsof the imaging devices 1A and 1B. Thus, the two images can be visuallyrecognized at the same time.

In the examples described in the first, third, fourth, fifth, sixth, andseventh embodiments, the image combining unit 12 sets the layoutposition of the enlarged image 22 in the peripheral field image 20 so asto follow the position of the imaging angle-of-view frame 21 whilemaintaining a predetermined distance relationship.

In the peripheral field image 20, the enlarged image 22 follows theimaging angle-of-view frame 21 while maintaining a predetermineddistance relationship, so that the enlarged image 22 and the imagingangle-of-view frame 21 do not overlap each other, and are disposed onthe display while maintaining a certain distance relationship. Thus, theuser can easily view both the enlarged image 22 and the imagingangle-of-view frame 21.

In the examples described in the first, third, fourth, fifth, sixth, andseventh embodiments, the image combining unit 12 sets the layoutposition of the enlarged image 22 in the peripheral field image 20 so asto follow changes in the position of the imaging angle-of-view frame 21in a horizontal direction while maintaining a predetermined horizontaldistance.

In the peripheral field image 20, the enlarged image 22 follows theimaging angle-of-view frame 21 while maintaining a predetermineddistance relationship in a horizontal direction. Thus, the user canvisually recognize both the enlarged image 22 and the imagingangle-of-view frame 21, with a constant distance being kept in betweenin a horizontal direction.

In the example described in the fifth embodiment, the image combiningunit 12 sets the layout position of the enlarged image 22 in theperipheral field image 20 so as to follow changes in the position of theimaging angle-of-view frame in a vertical direction while maintaining apredetermined vertical distance.

In the peripheral field image 20, the enlarged image 22 follows theimaging angle-of-view frame 21 while maintaining a predetermineddistance relationship in a vertical direction. Thus, the user canvisually recognize both the enlarged image 22 and the imagingangle-of-view frame 21, with a constant distance being kept in betweenin a vertical direction.

In the examples described above in the first, third, fourth, fifth,sixth, and seventh embodiments, the image combining unit 12 calculatesthe coordinate values of the layout position of the enlarged image 22 inthe coordinate space of the peripheral field image 20 by an arithmeticoperation using the coordinate values of the imaging angle-of-view frame21.

For example, the coordinates of the layout position of the enlargedimage 22 that maintains a predetermined distance from the imagingangle-of-view frame 21 are calculated using the coordinate values of thecenter and four corners of the imaging angle-of-view frame.

Thus, the layout position coordinate values of the enlarged image 22 canbe obtained by a simple arithmetic operation.

In the examples described above in the first, third, fourth, and fifthembodiments, the image combining unit 12 divides the inside of theperipheral field image 20 into a plurality of areas, and sets the layoutposition of the enlarged image 22 by a calculation formula selected inaccordance with the area in which the imaging angle-of-view frame 21 islocated.

In the peripheral field image 20, both the enlarged image 22 and theimaging angle-of-view frame 21 are preferably displayed at appropriatepositions. However, if the enlarged image 22 simply follows the imagingangle-of-view frame 21, both images are too close to the right or theleft, or the enlarged image 22 cannot be displayed, for example.Therefore, the peripheral field image 20 is divided into areas in ahorizontal direction and an upper limit direction, and the layoutposition of the enlarged image 22 is set using different calculationformulas for the respective areas so that the layout position isappropriately set for each area. With this arrangement, regardless ofthe position of the imaging angle-of-view frame 21, the user canvisually recognize the enlarged image 22 and the imaging angle-of-viewframe 21 at appropriate positions in the peripheral field image 20.

Note that FIG. 10 illustrates a case where the peripheral field image ishorizontally divided, and FIG. 19 illustrates a case where theperipheral field image is vertically and horizontally divided. However,there may be a case where the peripheral field image is verticallydivided. The number of divided areas, the size of each area, and thelike may also have various values.

In the examples described above in the third and fifth embodiments, theplurality of areas obtained by dividing the peripheral field image 20includes a buffer area for maintaining a calculation formula selectedstate.

For example, the area AR2 in the third embodiment, and the areas AR2,AR5, and AR8 in the fifth embodiment correspond to the buffer area.These areas maintain the previous calculation formulas (see step S172 inFIG. 17 , and steps S212, S222, and S232 in FIG. 20 ).

Such a buffer area that maintains the calculation formula selected stateprevents frequent switching of calculation formulas to be used in eachframe, for example. Thus, a chattering phenomenon in which the displayposition of the enlarged image 22 is frequently switched in a horizontaldirection is prevented, and a preferable viewing state for the user canbe provided.

In the examples described above in the sixth and seventh embodiments,the image combining unit 12 sets the layout position of the enlargedimage 22 in the peripheral field image 20 in accordance with theposition of the imaging angle-of-view frame 21. Also, in a case wherethe set layout position is such that all or part of the enlarged image22 is located outside the range of the peripheral field image 20, theimage combining unit 12 again sets the layout position of the enlargedimage 22 so that the imaging direction of the enlarged image 22 relativeto the imaging angle-of-view frame 21 is changed.

In the peripheral field image 20, when the layout position of theenlarged image 22 is set in accordance with the imaging angle-of-viewframe 21, the enlarged image 22 might have a portion outside the displayrange of the peripheral field image 20. In that case, the layoutposition of the enlarged image 22 is horizontally inverted relative tothe imaging angle-of-view frame 21. That is, the calculation formulasare changed, and recalculation is performed (see step S131 in FIG. 24 ,step S302 in FIG. 25 , and FIGS. 26 to 30 ). Thus, regardless of theposition of the imaging angle-of-view frame 21, the user can visuallyrecognize the enlarged image 22 in the peripheral field image 20 withoutany undisplayed portion, the enlarged image 22 being located at anappropriate distance from the imaging angle-of-view frame 21.

Note that, as well as or instead of the horizontal inversion, verticalinversion or position change may be caused.

In the examples described above in the sixth and seventh embodiments,the image combining unit 12 sets the layout position of the enlargedimage 22 in the peripheral field image 20 in accordance with theposition of the imaging angle-of-view frame 21. Also, in a case wherethe layout position is such that all or part of the enlarged image 22 islocated outside the range of the peripheral field image 20, the imagecombining unit 12 again sets the layout position of the enlarged image22 so that the layout position of the enlarged image 22 shifts towardthe range of the peripheral field image 20.

In the peripheral field image 20, when the layout position of theenlarged image 22 is made to follow the imaging angle-of-view frame 21,the enlarged image 22 might have a portion outside the display range ofthe peripheral field image 20. In that case, it might be possible toeliminate the outside portion by causing the layout position of theenlarged image 22 to shift toward the region in which outside portionsare prevented from appearing (see step S131 in FIG. 24 , and step S304in FIG. 25 ). With this arrangement, regardless of the position of theimaging angle-of-view frame 21, the user can visually recognize theenlarged image 22 in the peripheral field image 20 without anyundisplayed portion.

Although the cases where a horizontal shift, or horizontal and verticalshifts are performed have been described, there may of course beexamples in which only a vertical shift is performed.

In the examples described above in the first, third, fourth, fifth,sixth, and seventh embodiments, the image combining unit 12 sets thelayout position of the enlarged image 22 in the peripheral field image20 in accordance with the position of the imaging angle-of-view frame21. Also, in a case where the set layout position is such that all orpart of the enlarged image 22 is located outside the range of theperipheral field image 20, a combining process is not performed on theenlarged image 22.

In the peripheral field image 20, when the layout position of theenlarged image 22 is made to follow the imaging angle-of-view frame 21,the enlarged image 22 might have a portion outside the display range ofthe peripheral field image 20, and there are even cases where theoutside portion cannot be corrected. In such a case, the enlarged image22 is not combined, and the enlarged image 22 is not displayed (seesteps S106 and S110 in FIG. 6 , and step S303 in FIG. 25 or step S323 inFIG. 30 to step S110 in FIG. 24 ).

As a result, when the enlarged image 22 cannot be appropriatelydisplayed, the enlarged image 22 is not displayed, and thus, it ispossible to prevent the user from visually recognizing the enlargedimage in an inappropriate state.

In the examples described above in the first to seventh embodiments, ina case where the range detection unit 10 detects the imaging device 1Bin the peripheral field image 20, such as a case where a state in whichthe imaging device 1B is captured in the image is detected, for example,control is performed to output the enlarged image 22 as a display image.

In a case where the back surface of the imaging device 1B is captured inthe peripheral field image 20, there is a high possibility that theentire image of the field range of the imaging device 1B is not capturedin the peripheral field image 20. That is, the imaging device 1B becomesan obstacle, and the scene being imaged by the imaging device 1B doesnot appear in the peripheral field image 20. In such a case, the imagebeing captured by imaging device 1B is directly displayed as theenlarged image 22 (see steps S120, S121, and S122 in FIGS. 6 and 24).

The user does not visually recognize an image blocked by the imagingdevice 1B in the peripheral field image 20.

Further, if the user wears the imaging device 1A as a goggle camera asshown in FIG. 2 , for example, the state shown in FIG. 2B is like abehavior of the user holding the imaging device 1B in front of the eyesand looking into the viewfinder in a normal manner. Therefore, when theimage being captured by the imaging device 1B in that state isdisplayed, not only any feeling of strangeness is caused, but also astate in which the user is looking into the viewfinder is reproduced.Thus, this is also suitable for the user's imaging operation.

Note that, in some configuration, the control for outputting theenlarged image may be performed by the image combining unit 12, or theimage captured by the imaging device 1B may be output to the displaydevice 2 without passing through the image combining unit 12.Accordingly, the functions as an output control unit that outputs theenlarged image 22 may be formed by the image combining unit 12 or someother functional module in the image processing device 3.

In the examples described above in the first to seventh embodiments, ina case where the range detection unit 10 cannot detect the range of theimaging field of view of the imaging device 1B in the peripheral fieldimage 20, the image combining unit 12 sets the layout position of theenlarged image 22 at a preset fixed position in the peripheral fieldimage 20.

In a case where the display position of the imaging angle-of-view frame21 cannot be detected because the range of the imaging field of view ofthe imaging device 1B does not exist in the peripheral field image 20,the enlarged image 22 is combined and displayed at a specific positionin the peripheral field image 20 (see step S123 in FIGS. 6 and 24 ). Forexample, such a state occurs in the situation illustrated in FIG. 2C.

In this case, the enlarged image 22 is displayed at a specific positionin the peripheral field image 20, so that the user can check the imageof the imaging field of view of the imaging device 1B outside theperipheral field image 20. Particularly, as the imaging angle-of-viewframe 21 cannot be displayed in this case, the imaging angle-of-viewframe 21 and the enlarged image 22 do not overlap each other, andvisibility is not degraded.

Note that the specific position at which the enlarged image 22 isdisplayed may be a fixed position at all times, but the user may beallowed to designate any desired position.

In the examples described above in the first to seventh embodiments, ina case where the range detection unit 10 cannot detect the range of theimaging field of view of the imaging device 1B in the peripheral fieldimage 20, control is performed to output the peripheral field image 20as a display image.

In a case where the display position of the imaging angle-of-view frame21 cannot be detected because the range of the imaging field of view ofthe imaging device 1B does not exist in the peripheral field image 20,the peripheral field image 20 with which the enlarged image 22 is notcombined may be displayed (see step S123 in FIGS. 6 and 24 ). That is,the enlarged image 22 is hidden.

For example, if the imaging device 1A as a goggle camera is worn asshown in FIG. 2 , it is most often the case that the user is not awareof the image of the imaging device 1B while the imaging field-of-viewdirections of the imaging devices 1A and 1B are completely different asshown in FIG. 2B. Therefore, it is also preferable to make the entireperipheral field image 20 easy to view, by not displaying the enlargedimage 22.

Note that, in the process in step S123, the user's selection operationmay switch between performing a process of combining and displaying theenlarged image 22 at the specific position, and not displaying theenlarged image 22.

Further, in some configuration, the control for outputting the enlargedimage may be performed by the image combining unit 12, or the imagecaptured by the imaging device 1A may be output to the display device 2without passing through the image combining unit 12. Accordingly, thefunctions as an output control unit that outputs the peripheral fieldimage 20 may be formed by the image combining unit 12 or some otherfunctional module in the image processing device 3.

In the examples described above in the first, third, fourth, fifth,sixth, and seventh embodiments, the image combining unit 12 determineswhether or not to combine the enlarged image 22 with the peripheralfield image 20, in accordance with the size of the imaging angle-of-viewframe 21.

In a case where the imaging device 1B performs imaging at a wide angleand the imaging angle-of-view frame 21 is larger than the enlarged image22, or in a case where the imaging angle-of-view frame 21 is large tosome extent not in comparison with the enlarged image 22, there isalmost no point displaying the enlarged image 22. This is because theuser can check the imaging field of view of imaging device 1B in detailas the range surrounded by the imaging angle-of-view frame 21. Further,when the imaging angle-of-view frame 21 becomes larger, it becomesdifficult to appropriately dispose and display the enlarged image 22together with the imaging angle-of-view frame 21, and the enlarged image22 might become an obstacle instead.

Therefore, in a case where the size of the imaging angle-of-view frame21 is larger than the size of the enlarged image 22, for example, theenlarged image 22 is not displayed (see steps S104 and S110 in FIGS. 6and 24 ). Alternatively, in a case where the size of the imagingangle-of-view frame 21 is equal to or larger than a predetermined size,the enlarged image 22 may not be displayed.

As a result, while the user can check the imaging field of view of theimaging device 1B with the imaging angle-of-view frame 21, it ispossible to prevent the visibility from deteriorating due to the displayof the enlarged image 22, and to make the peripheral field of vieweasier to view.

In the examples described above in the first, third, fourth, fifth,sixth, and seventh embodiments, the imaging angle-of-view framecombining unit 11 combines the non-rectangular imaging angle-of-viewframes 21 with the peripheral field image 20 (see FIG. 32 ).

For example, by displaying the imaging angle-of-view frame 21 as aprojection range or the like from the viewpoint E of the imaging device1B, the field of view of the imaging device 1B can be stereoscopicallychecked in the peripheral field image 20.

The example described above in the second embodiment includes the rangedetection unit 10 that detects the range of the focus frame of theimaging device 1B in the peripheral field image 20, and the focus framecombining unit 11A that combines the peripheral field image 20 with thefocus frame indicating the focusing region detected by the rangedetection unit 10. The image combining unit 12 sets the layout positionof the enlarged image 22 in the peripheral field image 20 in accordancewith the position of the focus frame 23.

As the focus frame 23 is displayed in the peripheral field image 20, itis possible to check on which range the imaging device 1B is focusingwhile imaging in the peripheral field image 20, and the user can moreeasily follow the target object.

Further, as the imaging angle-of-view frame 21 and the enlarged image 22are displayed, it is possible to check the image of the target object(the detailed situation of the target object) being captured by theimaging device 1B, while checking the position of the target object inthe peripheral field image 20.

Further, in this case, it is possible to perform position setting suchthat the focus frame 23 and the enlarged image 22 do not overlap eachother, and the visibility is improved.

Note that, in the example in which the focus frame 23 is displayed, theprocesses related to the imaging angle-of-view frame 21 in the first,third, fourth, fifth, sixth, and seventh embodiments can be directlyapplied to the focus frame 23 and performed. That is, in a case wherethe focus frame 23 is displayed, the processes related to the positionof the enlarged image 22 according to each embodiment can be applied.

Note that, although the imaging devices 1A and 1B have been described asseparate imaging devices in the above embodiments, a first imagingsystem and a second imaging system may be provided as one imagingdevice. For example, the processes according to an embodiment can beadopted in an imaging device in which the first imaging system includesa wide-angle lens and captures the peripheral field image 20, and thesecond imaging system captures the target object with a narrower angleof view.

The program according to an embodiment is a program for causing a CPU, aDSP, a device including these components, or the like to perform any oneor a plurality of processes illustrated in FIGS. 6, 7, 16, 17, 20, 24,25, and 30 , for example.

That is, the program according to the embodiment is a program forcausing the image processing device 3 to perform a combining process ofdisposing the enlarged image 22, which is an image of the imaging fieldof view of the imaging device 1B capable of imaging a partial field ofview in the imaging field of view of the imaging device 1A, in theperipheral field image 20 captured by the imaging device 1A.

With such a program, the image processing device 3 described above canbe formed.

Such a program can be recorded beforehand in an HDD as a recordingmedium in a device such as a computer device, a ROM in a microcomputerincluding a CPU, or the like.

Alternatively, the program can be temporarily or permanently stored(recorded) in a removable recording medium, such as a flexible disk, acompact disc read only memory (CD-ROM), a magnet-optical (MO) disk, adigital versatile disc (DVD), a Blu-ray Disc (registered trademark), amagnetic disk, a semiconductor memory, or a memory card. Such aremovable recording medium can be provided as so-called packagedsoftware.

Alternatively, such a program can be installed from a removablerecording medium into a personal computer or the like, or can bedownloaded from a download site via a network such as a local areanetwork (LAN) or the Internet.

Also, such a program is suitable for providing image processing devices3 according to the embodiments in a wide range. For example, bydownloading the program into a personal computer, a portable informationprocessing device, a mobile telephone, a game device, a video device, apersonal digital assistant (PDA), or the like, the personal computer orthe like can be made to function as an image processing device 3 of thepresent disclosure.

Note that the advantageous effects described in this specification aremerely examples, and the advantageous effects of the present technologyare not limited to them and may include some other effects.

Note that the present technology can also be embodied in theconfigurations described below.

(1)

An image processing device including

an image combining unit that performs a combining process of disposing asecond image in a first image captured by a first imaging device, thesecond image being an image of an imaging field of view of a secondimaging device capable of imaging a partial field of view in the imagingfield of view of the first imaging device.

(2)

The image processing device according to (1), further including:

a range detection unit that detects a range of the imaging field of viewof the second imaging device in the first image; and

an imaging angle-of-view frame combining unit that combines an imagingangle-of-view frame with the first image, the imaging angle-of-viewframe indicating the range of the imaging field of view detected by therange detection unit.

(3)

The image processing device according to (2), in which

the image combining unit sets a layout position of the second image inthe first image, in accordance with a position of the imagingangle-of-view frame.

(4)

The image processing device according to (2) or (3), in which

the image combining unit sets a layout position of the second image inthe first image so as to follow the position of the imagingangle-of-view frame, with a predetermined distance relationship beingmaintained.

(5)

The image processing device according to (4), in which

the image combining unit sets the layout position of the second image inthe first image so as to follow a horizontal change in the position ofthe imaging angle-of-view frame while maintaining a predeterminedhorizontal distance.

(6)

The image processing device according to (4) of (5), in which

the image combining unit sets the layout position of the second image inthe first image so as to follow a horizontal change in the position ofthe imaging angle-of-view frame while maintaining a predeterminedvertical distance.

(7)

The image processing device according to any one of (4) to (6), in which

the image combining unit calculates a coordinate value of the layoutposition of the second image in a coordinate space of the first image byan arithmetic operation using a coordinate value of the imagingangle-of-view frame.

(8)

The image processing device according to any one of (2) to (7), in which

the image combining unit performs segmentation setting to divide thefirst image into a plurality of areas, and sets a layout position of thesecond image by a calculation formula selected in accordance with anarea in which the imaging angle-of-view frame is located.

(9)

The image processing device according to (8), in which the plurality ofareas includes a buffer area that maintains a state in which thecalculation formula is selected.

(10)

The image processing device according to any one of (2) to (9), in which

the image combining unit sets a layout position of the second image inthe first image in accordance with a position of the imagingangle-of-view frame, and, when the set layout position is such that allor part of the second image is located outside a range of the firstimage, the image combining unit again sets the layout position of thesecond image to change a direction of disposing the second imagerelative to the imaging angle-of-view frame.

(11)

The image processing device according to any one of (2) to (10), inwhich

the image combining unit sets a layout position of the second image inthe first image in accordance with a position of the imagingangle-of-view frame, and, when the set layout position is such that allor part of the second image is located outside a range of the firstimage, the image combining unit again sets the layout position of thesecond image to cause the layout position of the second image to shifttoward the range of the first image.

(12)

The image processing device according to any one of (2) to (11), inwhich

the image combining unit sets a layout position of the second image inthe first image in accordance with a position of the imagingangle-of-view frame, and, when the set layout position is such that allor part of the second image is located outside a range of the firstimage, the combining process is not performed on the second image.

(13)

The image processing device according to any one of (2) to (12), furtherincluding

an output control unit that performs control to output the second imageas a display image,

when the range detection unit has detected the second imaging device inthe first image.

(14)

The image processing device according to any one of (2) to (13), inwhich,

when the range detection unit fails to detect the range of the imagingfield of view of the second imaging device in the first image,

the image combining unit sets a layout position of the second image at apreset fixed position in the first image.

(15)

The image processing device according to any one of (2) to (13), furtherincluding

an output control unit that performs control to output the first imageas a display image,

when the range detection unit fails to detect the range of the imagingfield of view of the second imaging device in the first image.

(16)

The image processing device according to any one of (2) to (15), inwhich

the image combining unit determines whether or not to combine the secondimage with the first image, in accordance with a size of the imagingangle-of-view frame.

(17)

The image processing device according to any one of (2) to (16), inwhich

the imaging angle-of-view frame combining unit combines the imagingangle-of-view frame that is non-rectangular in shape with the firstimage.

(18)

The image processing device according to (1), further including:

a range detection unit that detects a range of a focus frame of thesecond imaging device in the first image; and

a focus frame combining unit that combines the focus frame with thefirst image, the focus frame indicating a focusing region detected bythe range detection unit,

in which

the image combining unit sets a layout position of the second image inthe first image in accordance with a position of the focus frame.

(19)

An image processing method implemented in an image processing device,the image processing method including

performing a combining process of disposing a second image in a firstimage captured by a first imaging device, the second image being animage of an imaging field of view of a second imaging device capable ofimaging a partial field of view in the imaging field of view of thefirst imaging device.

(20)

A program for causing an image processing device to perform

a combining process of disposing a second image in a first imagecaptured by a first imaging device, the second image being an image ofan imaging field of view of a second imaging device capable of imaging apartial field of view in the imaging field of view of the first imagingdevice.

REFERENCE SIGNS LIST

-   1A, 1B Imaging device-   2 Display device-   3 Image processing device-   4 Imaging unit-   10 Range detection unit-   11 Imaging angle-of-view frame combining unit-   11A Focus frame combining unit-   12 Image combining unit-   13 Display control unit-   20 Peripheral field image-   21 Imaging angle-of-view frame-   22 Enlarged image-   23 Focus frame-   32 Imaging unit-   33 Image signal processing unit-   34 Image analysis unit-   35 Control unit-   36 Operation unit-   37 Sensor unit-   38 Display control unit-   39 Display unit-   40 Storage unit-   41 External input unit

1. An image processing device comprising an image combining unit thatperforms a combining process of disposing a second image in a firstimage captured by a first imaging device, the second image being animage of an imaging field of view of a second imaging device capable ofimaging a partial field of view in an imaging field of view of the firstimaging device.
 2. The image processing device according to claim 1,further comprising: a range detection unit that detects a range of theimaging field of view of the second imaging device in the first image;and an imaging angle-of-view frame combining unit that combines animaging angle-of-view frame with the first image, the imagingangle-of-view frame indicating the range of the imaging field of viewdetected by the range detection unit.
 3. The image processing deviceaccording to claim 2, wherein the image combining unit sets a layoutposition of the second image in the first image, in accordance with aposition of the imaging angle-of-view frame.
 4. The image processingdevice according to claim 2, wherein the image combining unit sets alayout position of the second image in the first image to follow aposition of the imaging angle-of-view frame, with a predetermineddistance relationship being maintained.
 5. The image processing deviceaccording to claim 4, wherein the image combining unit sets the layoutposition of the second image in the first image to follow a horizontalchange in the position of the imaging angle-of-view frame whilemaintaining a predetermined horizontal distance.
 6. The image processingdevice according to claim 4, wherein the image combining unit sets thelayout position of the second image in the first image to follow avertical change in the position of the imaging angle-of-view frame whilemaintaining a predetermined vertical distance.
 7. The image processingdevice according to claim 4, wherein the image combining unit calculatesa coordinate value of the layout position of the second image in acoordinate space of the first image by an arithmetic operation using acoordinate value of the imaging angle-of-view frame.
 8. The imageprocessing device according to claim 2, wherein the image combining unitperforms segmentation setting to divide the first image into a pluralityof areas, and sets a layout position of the second image by acalculation formula selected in accordance with an area in which theimaging angle-of-view frame is located.
 9. The image processing deviceaccording to claim 8, wherein the plurality of areas includes a bufferarea that maintains a state in which the calculation formula isselected.
 10. The image processing device according to claim 2, whereinthe image combining unit sets a layout position of the second image inthe first image in accordance with a position of the imagingangle-of-view frame, and, when the set layout position is such that allor part of the second image is located outside a range of the firstimage, the image combining unit again sets the layout position of thesecond image to change a direction of disposing the second imagerelative to the imaging angle-of-view frame.
 11. The image processingdevice according to claim 2, wherein the image combining unit sets alayout position of the second image in the first image in accordancewith a position of the imaging angle-of-view frame, and, when the setlayout position is such that all or part of the second image is locatedoutside a range of the first image, the image combining unit again setsthe layout position of the second image to cause the layout position ofthe second image to shift toward the range of the first image.
 12. Theimage processing device according to claim 2, wherein the imagecombining unit sets a layout position of the second image in the firstimage in accordance with a position of the imaging angle-of-view frame,and, when the set layout position is such that all or part of the secondimage is located outside a range of the first image, the combiningprocess is not performed on the second image.
 13. The image processingdevice according to claim 2, further comprising an output control unitthat performs control to output the second image as a display image,when the range detection unit has detected the second imaging device inthe first image.
 14. The image processing device according to claim 2,wherein, when the range detection unit fails to detect the range of theimaging field of view of the second imaging device in the first image,the image combining unit sets a layout position of the second image at apreset fixed position in the first image.
 15. The image processingdevice according to claim 2, further comprising an output control unitthat performs control to output the first image as a display image, whenthe range detection unit fails to detect the range of the imaging fieldof view of the second imaging device in the first image.
 16. The imageprocessing device according to claim 2, wherein the image combining unitdetermines whether or not to combine the second image with the firstimage, in accordance with a size of the imaging angle-of-view frame. 17.The image processing device according to claim 2, wherein the imagingangle-of-view frame combining unit combines the imaging angle-of-viewframe that is non-rectangular in shape with the first image.
 18. Theimage processing device according to claim 1, further comprising: arange detection unit that detects a range of a focus frame of the secondimaging device in the first image; and a focus frame combining unit thatcombines the focus frame with the first image, the focus frameindicating a focusing region detected by the range detection unit,wherein the image combining unit sets a layout position of the secondimage in the first image in accordance with a position of the focusframe.
 19. An image processing method implemented in an image processingdevice, the image processing method comprising performing a combiningprocess of disposing a second image in a first image captured by a firstimaging device, the second image being an image of an imaging field ofview of a second imaging device capable of imaging a partial field ofview in an imaging field of view of the first imaging device.
 20. Aprogram for causing an image processing device to perform a combiningprocess of disposing a second image in a first image captured by a firstimaging device, the second image being an image of an imaging field ofview of a second imaging device capable of imaging a partial field ofview in an imaging field of view of the first imaging device.